Heterogeneous memory management (mirror process address space on a device mmu).
Subsystems (29)
In a nutshell:
The heterogeneous memory management (hmm) patchset implement a new api that sit on top of the mmu notifier api. It provides a simple api to device driver to mirror a process address space without having to lock or take reference on page and block them from being reclam or migrated. Any changes on a process address space is mirrored to the device page table by the hmm code. To achieve this not only we need each driver to implement a set of callback functions but hmm also interface itself in many key location of the mm code and fs code. Moreover hmm allow to migrate range of memory to the device remote memory to take advantages of its lower latency and higher bandwidth.
The why:
We want to be able to mirror a process address space so that compute api such as OpenCL or other similar api can start using the exact same address space on the GPU as on the CPU. This will greatly simplify usages of those api. Moreover we believe that we will see more and more specialize unit functions that will want to mirror process address using their own mmu.
To achieve this hmm requires :
A.1 - Hardware requirements
A.2 - sleeping inside mmu_notifier
A.3 - context information for mmu_notifier callback (patch 1 and 2) A.4 - new helper function for memcg (patch 5)
A.5 - special swap type and fault handling code
A.6 - file backed memory and filesystem changes
A.7 - The write back expectation
While avoiding :
B.1 - No new page flag
B.2 - No special page reclamation code
Finally the rest of this email deals with :
C.1 - Alternative designs
C.2 - Hardware solution
C.3 - Routines marked EXPORT_SYMBOL
C.4 - Planned features
C.5 - Getting upstream
But first patchlist :
0001 - Clarify the use of TTU_UNMAP as being done for VMSCAN or POISONING 0002 - Give context information to mmu_notifier callback ie why the callback
is made for (because of munmap call, or page migration, ...).
0003 - Provide the vma for which the invalidation is happening to mmu_notifier
callback. This is mostly and optimization to avoid looking up again the vma inside the mmu_notifier callback.
0004 - Add new helper to the generic interval tree (which use rb tree). 0005 - Add new helper to memcg so that anonymous page can be accounted as well
as unaccounted without a page struct. Also add a new helper function to transfer a charge to a page (charge which have been accounted without a struct page in the first place).
0006 - Introduce the hmm basic code to support simple device mirroring of the
address space. It is fully functional modulo some missing bit (guard or huge page and few other small corner cases).
0007 - Introduce support for migrating anonymous memory to device memory. This
involve introducing a new special swap type and teach the mm page fault code about hmm.
0008 - Introduce support for migrating shared or private memory that is backed
by a file. This is way more complex than anonymous case as it needs to synchronize with and exclude other kernel code path that might try to access those pages.
0009 - Add hmm support to ext4 filesystem.
0010 - Introduce a simple dummy driver that showcase use of the hmm api. 0011 - Add support for remote memory to the dummy driver.
I believe that patch 1, 2, 3 are use full on their own as they could help fix some kvm issues (see https://lkml.org/lkml/2014/1/15/125) and they do not modify behavior of any current code (except that patch 3 might result in a larger number of call to mmu_notifier as many as there is different vma for a range).
Other patches have many rough edges but we would like to validate our design and see what we need to change before smoothing out any of them.
A.1 - Hardware requirements :
The hardware must have its own mmu with a page table per process it wants to mirror. The device mmu mandatory features are :
- per page read only flag.
- page fault support that stop/suspend hardware thread and support resuming those hardware thread once the page fault have been serviced.
- same number of bits for the virtual address as the target architecture (for instance 48 bits on current AMD 64).
Advanced optional features :
- per page dirty bit (indicating the hardware did write to the page).
- per page access bit (indicating the hardware did access the page).
A.2 - Sleeping in mmu notifier callback :
Because update device mmu might need to sleep, either for taking device driver lock (which might be consider fixable) or simply because invalidating the mmu might take several hundred millisecond and might involve allocating device or driver resources to perform the operation any of which might require to sleep.
Thus we need to be able to sleep inside mmu_notifier_invalidate_range_start at the very least. Also we need to call to mmu_notifier_change_pte to be bracketed by mmu_notifier_invalidate_range_start and mmu_notifier_invalidate_range_end. We need this because mmu_notifier_change_pte is call with the anon vma lock held (and this is a non sleepable lock).
A.3 - Context information for mmu_notifier callback :
There is a need to provide more context information on why a mmu_notifier call back does happen. Was it because userspace call munmap ? Or was it because the kernel is trying to free some memory ? Or because page is being migrated ?
The context is provided by using unique enum value associated with call site of mmu_notifier functions. The patch here just add the enum value and modify each call site to pass along the proper value.
The context information is important for management of the secondary mmu. For instance on a munmap the device driver will want to free all resources used by that range (device page table memory). This could as well solve the issue that was discussed in this thread https://lkml.org/lkml/2014/1/15/125 kvm can ignore mmu_notifier_invalidate_range based on the enum value.
A.4 - New helper function for memcg :
To keep memory control working as expect with the introduction of remote memory we need to add new helper function so we can account anonymous remote memory as if it was backed by a page. We also need to be able to transfer charge from the remote memory to pages and we need to be able clear a page cgroup without side effect to the memcg.
The patchset currently does add a new type of memory resource but instead just account remote memory as local memory (struct page) is. This is done with the minimum amount of change to the memcg code. I believe they are correct.
It might make sense to introduce a new sub-type of memory down the road so that device memory can be included inside the memcg accounting but we choose to not do so at first.
A.5 - Special swap type and fault handling code :
When some range of address is backed by device memory we need cpu fault to be aware of that so it can ask hmm to trigger migration back to local memory. To avoid too much code disruption we do so by adding a new special hmm swap type that is special cased in various place inside the mm page fault code. Refer to patch 7 for details.
A.6 - File backed memory and filesystem changes :
Using remote memory for range of address backed by a file is more complex than anonymous memory. There are lot more code path that might want to access pages that cache a file (for read, write, splice, ...). To avoid disrupting the code too much and sleeping inside page cache look up we decided to add hmm support on a per filesystem basis. So each filesystem can be teach about hmm and how to interact with it correctly.
The design is relatively simple, the radix tree is updated to use special hmm swap entry for any page which is in remote memory. Thus any radix tree look up will find the special entry and will know it needs to synchronize itself with hmm to access the file.
There is however subtleties. Updating the radix tree does not guarantee that hmm is the sole user of the page, another kernel/user thread might have done a radix look up before the radix tree update.
The solution to this issue is to first update the radix tree, then lock each page we are migrating, then unmap it from all the process using it and setting its mapping field to NULL so that once we unlock the page all existing code will thought that the page was either truncated or reclaimed in both cases all existing kernel code path will eith perform new look and see the hmm special entry or will just skip the page. Those code path were audited to insure that their behavior and expected result are not modified by this.
However this does not insure us exclusive access to the page. So at first when migrating such page to remote memory we map it read only inside the device and keep the page around so that both the device copy and the page copy contain the same data. If the device wishes to write to this remote memory then it call hmm fault code.
To allow write on remote memory hmm will try to free the page, if the page can be free then it means hmm is the unique user of the page and the remote memory can safely be written to. If not then this means that the page content might still be in use by some other process and the device driver have to choose to either wait or use the local memory instead. So local memory page are kept as long as there are other user for them. We likely need to hookup some special page reclamation code to force reclaiming those pages after a while.
A.7 - The write back expectation :
We also wanted to preserve the writeback and dirty balancing as we believe this is an important behavior (avoiding dirty content to stay for too long inside remote memory without being write back to disk). To avoid constantly migrating memory back and forth we decided to use existing page (hmm keep all shared page around and never free them for the lifetime of rmem object they are associated with) as temporary writeback source. On writeback the remote memory is mapped read only on the device and copied back to local memory which is use as source for the disk write.
This design choice can however be seen as counter productive as it means that the device using hmm will see its rmem map read only for writeback and then will have to wait for writeback to go through. Another choice would be to forget writeback while memory is on the device and pretend page are clear but this would break fsync and similar API for file that does have part of its content inside some device memory.
Middle ground might be to keep fsync and alike working but to ignore any other writeback.
B.1 - No new page flag :
While adding a new page flag would certainly help to find a different design to implement the hmm feature set. We tried to only think about design that do not require such a new flag.
B.2 - No special page reclamation code :
This is one of the big issue, should be isolate pages that are actively use by a device from the regular lru to a specific lru managed by the hmm code. In this patchset we decided to avoid doing so as it would just add complexity to already complex code.
Current code will trigger sleep inside vmscan when trying to reclaim page that belong to a process which is mirrored on a device. Is this acceptable or should we add a new hmm lru list that would handle all pages used by device in special way so that those pages are isolated from the regular page reclamation code.
C.1 - Alternative designs :
The current design is the one we believe provide enough ground to support all necessary features while keeping complexity as low as possible. However i think it is important to state that several others designs were tested and to explain why they were discarded.
D1) One of the first design introduced a secondary page table directly updated
by hmm helper functions. Hope was that this secondary page table could be in some way directly use by the device. That was naive ... to say the least.
D2) The secondary page table with hmm specific format, was another design that
we tested. In this one the secondary page table was not intended to be use by the device but was intended to serve as a buffer btw the cpu page table and the device page table. Update to the device page table would use the hmm page table.
While this secondary page table allow to track what is actively use and also gather statistics about it. It does require memory, in worst case as much as the cpu page table.
Another issue is that synchronization between cpu update and device trying to access this secondary page table was either prone to lock contention. Or was getting awfully complex to avoid locking all while duplicating complexity inside each of the device driver.
The killing bullet was however the fact that the code was littered with bug condition about discrepancy between the cpu and the hmm page table.
D3) Use a structure to track all actively mirrored range per process and per
device. This allow to have an exact view of which range of memory is in use by which device.
Again this need a lot of memory to track each of the active range and worst case would need more memory than a secondary page table (one struct range per page).
Issue here was with the complexity or merging and splitting range on address space changes.
D4) Use a structure to track all active mirrored range per process (shared by
all the devices that mirror the same process). This partially address the memory requirement of D3 but this leave the complexity of range merging and splitting intact.
The current design is a simplification of D4 in which we only track range of memory for memory that have been migrated to device memory. So for any others operations hmm directly access the cpu page table and forward the appropriate information to the device driver through the hmm api. We might need to go back to D4 design or a variation of it for some of the features we want add.
C.2 - Hardware solution :
What hmm try to achieve can be partially achieved using hardware solution. Such hardware solution is part of PCIE specification with the PASID (process address space id) and ATS (address translation service). With both of this PCIE feature a device can ask for a virtual address of a given process to be translated into its corresponding physical address. To achieve this the IOMMU bridge is capable of understanding and walking the cpu page table of a process. See the IOMMUv2 implementation inside the linux kernel for reference.
There is two huge restriction with hardware solution to this problem. First an obvious one is that you need hardware support. While HMM also require hardware support on the GPU side it does not on the architecture side (no requirement on IOMMU, or any bridges that are between the GPU and the system memory). This is a strong advantages to HMM it only require hardware support to one specific part.
The second restriction is that hardware solution like IOMMUv2 does not permit migrating chunk of memory to the device local memory which means under-using hardware resources (all discrete GPU comes with fast local memory that can have more than ten times the bandwidth of system memory).
This two reasons alone, are we believe enough to justify hmm usefulness.
Moreover hmm can work in a hybrid solution where non migrated chunk of memory goes through the hardware solution (IOMMUv2 for instance) and only the memory that is migrated to the device is handled by the hmm code. The requirement for the hardware is minimal, the hardware need to support the PASID & ATS (or any other hardware implementation of the same idea) on page granularity basis (it could be on the granularity of any level of the device page table so no need to populate all levels of the device page table). Which is the best solution for the problem.
C.3 - Routines marked EXPORT_SYMBOL
As these routines are intended to be referenced in device drivers, they are marked EXPORT_SYMBOL as is common practice. This encourages adoption of HMM in both GPL and non-GPL drivers, and allows ongoing collaboration with one of the primary authors of this idea.
I think it would be beneficial to include this feature as soon as possible. Early collaborators can go to the trouble of fixing and polishing the HMM implementation, allowing it to fully bake by the time other drivers start implementing features requiring it. We are confident that this API will be useful to others as they catch up with supporting hardware.
C.4 - Planned features :
We are planning to add various features down the road once we can clear the basic design. Most important ones are :
- Allowing inter-device migration for compatible devices.
- Allowing hmm_rmem without backing storage (simplify some of the driver).
- Device specific memcg.
- Improvement to allow APU to take advantages of rmem, by hiding the page from the cpu the gpu can use a different memory controller link that do not require cache coherency with the cpu and thus provide higher bandwidth.
- Atomic device memory operation by unmapping on the cpu while the device is performing atomic operation (this require hardware mmu to differentiate between regular memory access and atomic memory access and to have a flag that allow atomic memory access on per page basis).
- Pining private memory to rmem this would be a useful feature to add and would require addition of a new flag to madvise. Any cpu access would result in SIGBUS for the cpu process.
C.5 - Getting upstream :
So what should i do to get this patchset in a mergeable form at least at first as a staging feature ? Right now the patchset has few rough edges around huge page support and other smaller issues. But as said above i believe that patch 1, 2, 3 and 4 can be merge as is as they do not modify current behavior while being useful to other.
Should i implement some secondary hmm specific lru and their associated worker thread to avoid having the regular reclaim code to end up sleeping waiting for a device to update its page table ?
Should i go for a totaly different design ? If so what direction ? As stated above we explored other design and i listed there flaws.
Any others things that i need to fix/address/change/improve ?
Comments and flames are welcome.
Cheers,
Jérôme Glisse
To: <[email protected]>,
To: linux-mm <[email protected]>,
To: <[email protected]>,
Cc: "Mel Gorman" <[email protected]>,
Cc: "H. Peter Anvin" <[email protected]>,
Cc: "Peter Zijlstra" <[email protected]>,
Cc: "Andrew Morton" <[email protected]>,
Cc: "Linda Wang" <[email protected]>,
Cc: "Kevin E Martin" <[email protected]>,
Cc: "Jerome Glisse" <[email protected]>,
Cc: "Andrea Arcangeli" <[email protected]>,
Cc: "Johannes Weiner" <[email protected]>,
Cc: "Larry Woodman" <[email protected]>,
Cc: "Rik van Riel" <[email protected]>,
Cc: "Dave Airlie" <[email protected]>,
Cc: "Jeff Law" <[email protected]>,
Cc: "Brendan Conoboy" <[email protected]>,
Cc: "Joe Donohue" <[email protected]>,
Cc: "Duncan Poole" <[email protected]>,
Cc: "Sherry Cheung" <[email protected]>,
Cc: "Subhash Gutti" <[email protected]>,
Cc: "John Hubbard" <[email protected]>,
Cc: "Mark Hairgrove" <[email protected]>,
Cc: "Lucien Dunning" <[email protected]>,
Cc: "Cameron Buschardt" <[email protected]>,
Cc: "Arvind Gopalakrishnan" <[email protected]>,
Cc: "Haggai Eran" <[email protected]>,
Cc: "Or Gerlitz" <[email protected]>,
Cc: "Sagi Grimberg" <[email protected]>
Cc: "Shachar Raindel" <[email protected]>,
Cc: "Liran Liss" <[email protected]>,
Cc: "Roland Dreier" <[email protected]>,
Cc: "Sander, Ben" <[email protected]>,
Cc: "Stoner, Greg" <[email protected]>,
Cc: "Bridgman, John" <[email protected]>,
Cc: "Mantor, Michael" <[email protected]>,
Cc: "Blinzer, Paul" <[email protected]>,
Cc: "Morichetti, Laurent" <[email protected]>,
Cc: "Deucher, Alexander" <[email protected]>,
Cc: "Gabbay, Oded" <[email protected]>,
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/11 mm: differentiate unmap for vmscan from other unmap.
From: Jérôme Glisse <[email protected]>
New code will need to be able to differentiate between a regular unmap and an unmap trigger by vmscan in which case we want to be as quick as possible.
--- include/linux/rmap.h | 7 ++++--- mm/memory-failure.c | 2 +- mm/vmscan.c | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-)
include/linux/rmap.h +4 -3
@@ -72,9 +72,10 @@ struct anon_vma_chain { };enum ttu_flags { -TTU_UNMAP =0 ,/* unmap mode */ -TTU_MIGRATION =1 ,/* migration mode */ -TTU_MUNLOCK =2 ,/* munlock mode */ +TTU_VMSCAN =0 ,/* unmap for vmscan mode */ +TTU_POISON =1 ,/* unmap mode */ +TTU_MIGRATION =2 ,/* migration mode */ +TTU_MUNLOCK =3 ,/* munlock mode */ TTU_ACTION_MASK =0xff ,TTU_IGNORE_MLOCK = (1 <<8 ),/* ignore mlock */
mm/memory-failure.c +1 -1
@@ -854,7 +854,7 @@ static int page_action(struct page_state *ps, struct page *p,static int hwpoison_user_mappings (struct page *p,unsigned long pfn,int trapno,int flags,struct page **hpagep) { -enum ttu_flags ttu =TTU_UNMAP |TTU_IGNORE_MLOCK |TTU_IGNORE_ACCESS ; +enum ttu_flags ttu =TTU_POISON |TTU_IGNORE_MLOCK |TTU_IGNORE_ACCESS ;struct address_space *mapping;LIST_HEAD (tokill);int ret;
mm/vmscan.c +2 -2
@@ -1158,7 +1158,7 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone, } ret =shrink_page_list (&clean_pages, zone, &sc, -TTU_UNMAP |TTU_IGNORE_ACCESS , +TTU_VMSCAN |TTU_IGNORE_ACCESS , &dummy1, &dummy2, &dummy3, &dummy4, &dummy5, true);list_splice (&clean_pages, page_list);mod_zone_page_state (zone,NR_ISOLATED_FILE , -ret);
@@ -1511,7 +1511,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,if (nr_taken ==0 )return 0 ; - nr_reclaimed =shrink_page_list (&page_list, zone, sc,TTU_UNMAP , + nr_reclaimed =shrink_page_list (&page_list, zone, sc,TTU_VMSCAN , &nr_dirty, &nr_unqueued_dirty, &nr_congested, &nr_writeback, &nr_immediate, false);
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/11 mmu_notifier: add action information to address invalidation.
In reply to [email protected]From: Jérôme Glisse <[email protected]>
The action information will be usefull for new user of mmu_notifier API. The action argument differentiate between a vma disappearing, a page being write protected or simply a page being unmaped. This allow new user to take different action for instance on unmap the resource used to track a vma are still valid and should stay around if need be. While if the action is saying that a vma is being destroy it means that that any resources used to track this vma can be free.
--- drivers/iommu/amd_iommu_v2.c | 14 ++++-- drivers/xen/gntdev.c | 9 ++-- fs/proc/task_mmu.c | 4 +- include/linux/hugetlb.h | 4 +- include/linux/mmu_notifier.h | 108 ++++++++++++++++++++++++++++++++++--------- kernel/events/uprobes.c | 6 +-- mm/filemap_xip.c | 2 +- mm/fremap.c | 8 +++- mm/huge_memory.c | 26 +++++------ mm/hugetlb.c | 19 ++++---- mm/ksm.c | 12 ++--- mm/memory.c | 23 ++++----- mm/mempolicy.c | 2 +- mm/migrate.c | 6 +-- mm/mmu_notifier.c | 26 +++++++---- mm/mprotect.c | 30 ++++++++---- mm/mremap.c | 4 +- mm/rmap.c | 55 +++++++++++++++++++--- virt/kvm/kvm_main.c | 12 +++-- 19 files changed, 258 insertions(+), 112 deletions(-)
drivers/iommu/amd_iommu_v2.c +10 -4
@@ -421,21 +421,25 @@ static int mn_clear_flush_young(struct mmu_notifier *mn,static void mn_change_pte (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long address, -pte_t pte) +pte_t pte, +enum mmu_action action) {__mn_flush_page (mn, address); }static void mn_invalidate_page (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long address) +unsigned long address, +enum mmu_action action) {__mn_flush_page (mn, address); }static void mn_invalidate_range_start (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long start,unsigned long end) +unsigned long start, +unsigned long end, +enum mmu_action action) {struct pasid_state *pasid_state;struct device_state *dev_state;
@@ -449,7 +453,9 @@ static void mn_invalidate_range_start(struct mmu_notifier *mn,static void mn_invalidate_range_end (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long start,unsigned long end) +unsigned long start, +unsigned long end, +enum mmu_action action) {struct pasid_state *pasid_state;struct device_state *dev_state;
drivers/xen/gntdev.c +6 -3
@@ -428,7 +428,9 @@ static void unmap_if_in_range(struct grant_map *map,static void mn_invl_range_start (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long start,unsigned long end) +unsigned long start, +unsigned long end, +enum mmu_action action) { struct gntdev_priv *priv = container_of(mn, struct gntdev_priv, mn);struct grant_map *map;
@@ -445,9 +447,10 @@ static void mn_invl_range_start(struct mmu_notifier *mn,static void mn_invl_page (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long address) +unsigned long address, +enum mmu_action action) { -mn_invl_range_start (mn, mm, address, address +PAGE_SIZE ); +mn_invl_range_start (mn, mm, address, address +PAGE_SIZE , action); } static void mn_release(struct mmu_notifier *mn,
fs/proc/task_mmu.c +2 -2
@@ -818,11 +818,11 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, };down_read (&mm->mmap_sem);if (type ==CLEAR_REFS_SOFT_DIRTY ) -mmu_notifier_invalidate_range_start (mm,0 ,-1 ); +mmu_notifier_invalidate_range_start (mm,0 ,-1 ,MMU_SOFT_DIRTY );for (vma = mm->mmap; vma; vma = vma->vm_next)walk_page_vma (vma, &clear_refs_walk);if (type ==CLEAR_REFS_SOFT_DIRTY ) -mmu_notifier_invalidate_range_end (mm,0 ,-1 ); +mmu_notifier_invalidate_range_end (mm,0 ,-1 ,MMU_SOFT_DIRTY );flush_tlb_mm (mm);up_read (&mm->mmap_sem);mmput (mm);
include/linux/hugetlb.h +3 -1
@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/hugetlb_inline.h> #include <linux/cgroup.h> +#include <linux/mmu_notifier.h> #include <linux/list.h> #include <linux/kref.h>
@@ -103,7 +104,8 @@ struct page *follow_huge_pud(struct mm_struct *mm, unsigned long address,int pmd_huge (pmd_t pmd);int pud_huge (pud_t pmd);unsigned long hugetlb_change_protection (struct vm_area_struct *vma, -unsigned long address,unsigned long end,pgprot_t newprot); +unsigned long address,unsigned long end,pgprot_t newprot, +enum mmu_action action);#else /* !CONFIG_HUGETLB_PAGE */
include/linux/mmu_notifier.h +85 -23
201 lines not on this page · raw message
kernel/events/uprobes.c +3 -3
@@ -170,7 +170,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,/* For try_to_free_swap() and munlock_vma_page() below */ lock_page (page); -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_UNMAP ); err = -EAGAIN ; ptep =page_check_address (page, mm, addr, &ptl,0 );if (!ptep)
@@ -186,7 +186,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,flush_cache_page (vma, addr,pte_pfn (*ptep));ptep_clear_flush (vma, addr, ptep); -set_pte_at_notify (mm, addr, ptep,mk_pte (kpage, vma->vm_page_prot)); +set_pte_at_notify (mm, addr, ptep,mk_pte (kpage, vma->vm_page_prot),MMU_UNMAP );page_remove_rmap (page);if (!page_mapped (page))
@@ -199,7 +199,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, err =0 ; unlock: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_UNMAP );unlock_page (page);return err; }
mm/filemap_xip.c +1 -1
@@ -198,7 +198,7 @@ retry:BUG_ON (pte_dirty (pteval));pte_unmap_unlock (pte, ptl);/* must invalidate_page _before_ freeing the page */ -mmu_notifier_invalidate_page (mm, address); +mmu_notifier_invalidate_page (mm, address,MMU_UNMAP );page_cache_release (page); } }
mm/fremap.c +6 -2
@@ -254,9 +254,13 @@ get_write_lock: vma->vm_flags = vm_flags; } -mmu_notifier_invalidate_range_start (mm, start, start + size); +/* Consider it a ZAP operation for now, it could be seen as an unmap but +* remapping is trickier as it can change the vma to non linear and thus +* trigger side effect. +*/ +mmu_notifier_invalidate_range_start (mm, start, start + size,MMU_MUNMAP ); err = vma->vm_ops->remap_pages (vma, start, size, pgoff); -mmu_notifier_invalidate_range_end (mm, start, start + size); +mmu_notifier_invalidate_range_end (mm, start, start + size,MMU_MUNMAP );/* * We can't clear VM_NONLINEAR because we'd have to do
mm/huge_memory.c +13 -13
@@ -993,7 +993,7 @@ static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm, mmun_start = haddr; mmun_end = haddr +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); ptl =pmd_lock (mm, pmd);if (unlikely (!pmd_same (*pmd, orig_pmd)))
@@ -1023,7 +1023,7 @@ static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm,page_remove_rmap (page);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); ret |=VM_FAULT_WRITE ;put_page (page);
@@ -1033,7 +1033,7 @@ out: out_free_pages:spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP );mem_cgroup_uncharge_start ();for (i =0 ; i <HPAGE_PMD_NR ; i++) {mem_cgroup_uncharge_page (pages[i]);
@@ -1123,7 +1123,7 @@ alloc: mmun_start = haddr; mmun_end = haddr +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP );spin_lock (ptl);if (page)
@@ -1153,7 +1153,7 @@ alloc: }spin_unlock (ptl); out_mn: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); out:return ret; out_unlock:
@@ -1588,7 +1588,7 @@ static int __split_huge_page_splitting(struct page *page,const unsigned long mmun_start = address;const unsigned long mmun_end = address +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); pmd =page_check_address_pmd (page, mm, address,PAGE_CHECK_ADDRESS_PMD_NOTSPLITTING_FLAG , &ptl);if (pmd) {
@@ -1603,7 +1603,7 @@ static int __split_huge_page_splitting(struct page *page, ret =1 ;spin_unlock (ptl); } -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT );return ret; }
@@ -2402,7 +2402,7 @@ static void collapse_huge_page(struct mm_struct *mm, mmun_start = address; mmun_end = address +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); pmd_ptl =pmd_lock (mm, pmd);/* probably unnecessary */ /* * After this gup_fast can't run anymore. This also removes
@@ -2412,7 +2412,7 @@ static void collapse_huge_page(struct mm_struct *mm,*/ _pmd =pmdp_clear_flush (vma, address, pmd);spin_unlock (pmd_ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT );spin_lock (pte_ptl); isolated =__collapse_huge_page_isolate (vma, address, pte);
@@ -2801,24 +2801,24 @@ void __split_huge_page_pmd(struct vm_area_struct *vma, unsigned long address, mmun_start = haddr; mmun_end = haddr +HPAGE_PMD_SIZE ; again: -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); ptl =pmd_lock (mm, pmd);if (unlikely (!pmd_trans_huge (*pmd))) {spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT );return ; }if (is_huge_zero_pmd (*pmd)) {__split_huge_zero_page_pmd (vma, haddr, pmd);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT );return ; } page =pmd_page (*pmd);VM_BUG_ON_PAGE (!page_count (page), page);get_page (page);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT );split_huge_page (page);
mm/hugetlb.c +10 -9
@@ -2540,7 +2540,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, mmun_start = vma->vm_start; mmun_end = vma->vm_end;if (cow) -mmu_notifier_invalidate_range_start (src, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (src, mmun_start, mmun_end,MMU_COW );for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) {spinlock_t *src_ptl, *dst_ptl;
@@ -2574,7 +2574,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, }if (cow) -mmu_notifier_invalidate_range_end (src, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (src, mmun_start, mmun_end,MMU_COW );return ret; }
@@ -2626,7 +2626,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,BUG_ON (end & ~huge_page_mask (h));tlb_start_vma (tlb, vma); -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_UNMAP ); again:for (address = start; address < end; address += sz) { ptep =huge_pte_offset (mm, address);
@@ -2697,7 +2697,7 @@ unlock:if (address < end && !ref_page) goto again; } -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_UNMAP );tlb_end_vma (tlb, vma); }
@@ -2884,7 +2884,7 @@ retry_avoidcopy: mmun_start = address &huge_page_mask (h); mmun_end = mmun_start +huge_page_size (h); -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_UNMAP );/* * Retake the page table lock to check for racing updates * before the page tables are altered
@@ -2904,7 +2904,7 @@ retry_avoidcopy: new_page = old_page; }spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_UNMAP );page_cache_release (new_page);page_cache_release (old_page);
@@ -3329,7 +3329,8 @@ same_page: }unsigned long hugetlb_change_protection (struct vm_area_struct *vma, -unsigned long address,unsigned long end,pgprot_t newprot) +unsigned long address,unsigned long end,pgprot_t newprot, +enum mmu_action action) {struct mm_struct *mm = vma->vm_mm;unsigned long start = address;
@@ -3341,7 +3342,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,BUG_ON (address >= end);flush_cache_range (vma, address, end); -mmu_notifier_invalidate_range_start (mm, start, end); +mmu_notifier_invalidate_range_start (mm, start, end, action);mutex_lock (&vma->vm_file->f_mapping->i_mmap_mutex);for (; address < end; address +=huge_page_size (h)) {spinlock_t *ptl;
@@ -3371,7 +3372,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,*/ flush_tlb_range (vma, start, end);mutex_unlock (&vma->vm_file->f_mapping->i_mmap_mutex); -mmu_notifier_invalidate_range_end (mm, start, end); +mmu_notifier_invalidate_range_end (mm, start, end, action);return pages << h->order; }
mm/ksm.c +6 -6
@@ -872,7 +872,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, mmun_start = addr; mmun_end = addr +PAGE_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_KSM_RONLY ); ptep =page_check_address (page, mm, addr, &ptl,0 );if (!ptep)
@@ -904,7 +904,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page,if (pte_dirty (entry))set_page_dirty (page); entry =pte_mkclean (pte_wrprotect (entry)); -set_pte_at_notify (mm, addr, ptep, entry); +set_pte_at_notify (mm, addr, ptep, entry,MMU_KSM_RONLY ); } *orig_pte = *ptep; err =0 ;
@@ -912,7 +912,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, out_unlock:pte_unmap_unlock (ptep, ptl); out_mn: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_KSM_RONLY ); out:return err; }
@@ -949,7 +949,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, mmun_start = addr; mmun_end = addr +PAGE_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_KSM ); ptep =pte_offset_map_lock (mm, pmd, addr, &ptl);if (!pte_same (*ptep, orig_pte)) {
@@ -962,7 +962,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,flush_cache_page (vma, addr,pte_pfn (*ptep));ptep_clear_flush (vma, addr, ptep); -set_pte_at_notify (mm, addr, ptep,mk_pte (kpage, vma->vm_page_prot)); +set_pte_at_notify (mm, addr, ptep,mk_pte (kpage, vma->vm_page_prot),MMU_KSM );page_remove_rmap (page);if (!page_mapped (page))
@@ -972,7 +972,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,pte_unmap_unlock (ptep, ptl); err =0 ; out_mn: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_KSM ); out:return err; }
mm/memory.c +12 -11
@@ -1055,7 +1055,7 @@ int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, mmun_end = end;if (is_cow)mmu_notifier_invalidate_range_start (src_mm, mmun_start, - mmun_end); + mmun_end,MMU_COW ); ret =0 ; dst_pgd =pgd_offset (dst_mm, addr);
@@ -1072,7 +1072,8 @@ int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, }while (dst_pgd++, src_pgd++, addr = next, addr != end);if (is_cow) -mmu_notifier_invalidate_range_end (src_mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (src_mm, mmun_start, mmun_end, +MMU_COW );return ret; }
@@ -1378,10 +1379,10 @@ void unmap_vmas(struct mmu_gather *tlb, {struct mm_struct *mm = vma->vm_mm; -mmu_notifier_invalidate_range_start (mm, start_addr, end_addr); +mmu_notifier_invalidate_range_start (mm, start_addr, end_addr,MMU_MUNMAP );for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)unmap_single_vma (tlb, vma, start_addr, end_addr,NULL ); -mmu_notifier_invalidate_range_end (mm, start_addr, end_addr); +mmu_notifier_invalidate_range_end (mm, start_addr, end_addr,MMU_MUNMAP ); }/**
@@ -1403,10 +1404,10 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start,lru_add_drain ();tlb_gather_mmu (&tlb, mm, start, end);update_hiwater_rss (mm); -mmu_notifier_invalidate_range_start (mm, start, end); +mmu_notifier_invalidate_range_start (mm, start, end,MMU_MUNMAP );for ( ; vma && vma->vm_start < end; vma = vma->vm_next)unmap_single_vma (&tlb, vma, start, end, details); -mmu_notifier_invalidate_range_end (mm, start, end); +mmu_notifier_invalidate_range_end (mm, start, end,MMU_MUNMAP );tlb_finish_mmu (&tlb, start, end); }
@@ -1429,9 +1430,9 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addrlru_add_drain ();tlb_gather_mmu (&tlb, mm, address, end);update_hiwater_rss (mm); -mmu_notifier_invalidate_range_start (mm, address, end); +mmu_notifier_invalidate_range_start (mm, address, end,MMU_MUNMAP );unmap_single_vma (&tlb, vma, address, end, details); -mmu_notifier_invalidate_range_end (mm, address, end); +mmu_notifier_invalidate_range_end (mm, address, end,MMU_MUNMAP );tlb_finish_mmu (&tlb, address, end); }
@@ -2850,7 +2851,7 @@ gotten: mmun_start = address &PAGE_MASK ; mmun_end = mmun_start +PAGE_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_FAULT_WP );/* * Re-check the pte - we dropped the lock
@@ -2880,7 +2881,7 @@ gotten:* mmu page tables (such as kvm shadow page tables), we want the * new page to be mapped directly into the secondary page table. */ -set_pte_at_notify (mm, address, page_table, entry); +set_pte_at_notify (mm, address, page_table, entry,MMU_FAULT_WP );update_mmu_cache (vma, address, page_table);if (old_page) {/*
@@ -2919,7 +2920,7 @@ gotten: unlock:pte_unmap_unlock (page_table, ptl);if (mmun_end > mmun_start) -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_FAULT_WP );if (old_page) {/* * Don't let another task, with possibly unlocked vma,
mm/mempolicy.c +1 -1
@@ -561,7 +561,7 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, {int nr_updated; - nr_updated =change_protection (vma, addr, end, vma->vm_page_prot,0 ,1 ); + nr_updated =change_protection (vma, addr, end, vma->vm_page_prot,0 ,1 ,0 );if (nr_updated)count_vm_numa_events (NUMA_PTE_UPDATES , nr_updated);
mm/migrate.c +3 -3
@@ -1804,12 +1804,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,WARN_ON (PageLRU (new_page));/* Recheck the target PMD */ -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_MIGRATE ); ptl =pmd_lock (mm, pmd);if (unlikely (!pmd_same (*pmd, entry) ||page_count (page) !=2 )) { fail_putback:spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_MIGRATE );/* Reverse changes made by migrate_page_copy() */ if (TestClearPageActive (new_page))
@@ -1875,7 +1875,7 @@ fail_putback:*/ mem_cgroup_end_migration (memcg, page, new_page, true);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_MIGRATE );/* Take an "isolate" reference and put new page on the LRU. */ get_page (new_page);
mm/mmu_notifier.c +17 -9
@@ -122,8 +122,10 @@ int __mmu_notifier_test_young(struct mm_struct *mm,return young; } -void __mmu_notifier_change_pte (struct mm_struct *mm,unsigned long address, -pte_t pte) +void __mmu_notifier_change_pte (struct mm_struct *mm, +unsigned long address, +pte_t pte, +enum mmu_action action) {struct mmu_notifier *mn;int id;
@@ -131,13 +133,14 @@ void __mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, id =srcu_read_lock (&srcu);hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) {if (mn->ops->change_pte) - mn->ops->change_pte (mn, mm, address, pte); + mn->ops->change_pte (mn, mm, address, pte, action); }srcu_read_unlock (&srcu, id); }void __mmu_notifier_invalidate_page (struct mm_struct *mm, -unsigned long address) +unsigned long address, +enum mmu_action action) {struct mmu_notifier *mn;int id;
@@ -145,13 +148,16 @@ void __mmu_notifier_invalidate_page(struct mm_struct *mm, id =srcu_read_lock (&srcu);hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) {if (mn->ops->invalidate_page) - mn->ops->invalidate_page (mn, mm, address); + mn->ops->invalidate_page (mn, mm, address, action); }srcu_read_unlock (&srcu, id); }void __mmu_notifier_invalidate_range_start (struct mm_struct *mm, -unsigned long start,unsigned long end) +unsigned long start, +unsigned long end, +enum mmu_action action) + {struct mmu_notifier *mn;int id;
@@ -159,14 +165,16 @@ void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, id =srcu_read_lock (&srcu);hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) {if (mn->ops->invalidate_range_start) - mn->ops->invalidate_range_start (mn, mm, start, end); + mn->ops->invalidate_range_start (mn, mm, start, end, action); }srcu_read_unlock (&srcu, id); }EXPORT_SYMBOL_GPL (__mmu_notifier_invalidate_range_start);void __mmu_notifier_invalidate_range_end (struct mm_struct *mm, -unsigned long start,unsigned long end) +unsigned long start, +unsigned long end, +enum mmu_action action) {struct mmu_notifier *mn;int id;
@@ -174,7 +182,7 @@ void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, id =srcu_read_lock (&srcu);hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) {if (mn->ops->invalidate_range_end) - mn->ops->invalidate_range_end (mn, mm, start, end); + mn->ops->invalidate_range_end (mn, mm, start, end, action); }srcu_read_unlock (&srcu, id); }
mm/mprotect.c +21 -9
@@ -137,7 +137,8 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,static inline unsigned long change_pmd_range (struct vm_area_struct *vma,pud_t *pud,unsigned long addr,unsigned long end, -pgprot_t newprot,int dirty_accountable,int prot_numa) +pgprot_t newprot,int dirty_accountable,int prot_numa, +enum mmu_action action) {pmd_t *pmd;struct mm_struct *mm = vma->vm_mm;
@@ -157,7 +158,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,/* invoke the mmu notifier if the pmd is populated */ if (!mni_start) { mni_start = addr; -mmu_notifier_invalidate_range_start (mm, mni_start, end); +mmu_notifier_invalidate_range_start (mm, mni_start, end, action); }if (pmd_trans_huge (*pmd)) {
@@ -185,7 +186,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, }while (pmd++, addr = next, addr != end);if (mni_start) -mmu_notifier_invalidate_range_end (mm, mni_start, end); +mmu_notifier_invalidate_range_end (mm, mni_start, end, action);if (nr_huge_updates)count_vm_numa_events (NUMA_HUGE_PTE_UPDATES , nr_huge_updates);
@@ -194,7 +195,8 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,static inline unsigned long change_pud_range (struct vm_area_struct *vma,pgd_t *pgd,unsigned long addr,unsigned long end, -pgprot_t newprot,int dirty_accountable,int prot_numa) +pgprot_t newprot,int dirty_accountable,int prot_numa, +enum mmu_action action) {pud_t *pud;unsigned long next;
@@ -206,7 +208,7 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma,if (pud_none_or_clear_bad (pud))continue ; pages +=change_pmd_range (vma, pud, addr, next, newprot, - dirty_accountable, prot_numa); + dirty_accountable, prot_numa, action); }while (pud++, addr = next, addr != end);return pages;
@@ -214,7 +216,7 @@ static inline unsigned long change_pud_range(struct vm_area_struct *vma,static unsigned long change_protection_range (struct vm_area_struct *vma,unsigned long addr,unsigned long end,pgprot_t newprot, -int dirty_accountable,int prot_numa) +int dirty_accountable,int prot_numa,enum mmu_action action) {struct mm_struct *mm = vma->vm_mm;pgd_t *pgd;
@@ -231,7 +233,7 @@ static unsigned long change_protection_range(struct vm_area_struct *vma,if (pgd_none_or_clear_bad (pgd))continue ; pages +=change_pud_range (vma, pgd, addr, next, newprot, - dirty_accountable, prot_numa); + dirty_accountable, prot_numa, action); }while (pgd++, addr = next, addr != end);/* Only flush the TLB if we actually modified any entries: */
@@ -247,11 +249,21 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start, int dirty_accountable, int prot_numa) {unsigned long pages; +enum mmu_action action =MMU_MPROT_NONE ; + +/* At this points vm_flags is updated. */ +if ((vma->vm_flags &VM_READ ) && (vma->vm_flags &VM_WRITE )) { + action =MMU_MPROT_RANDW ; + }else if (vma->vm_flags &VM_WRITE ) { + action =MMU_MPROT_WONLY ; + }else if (vma->vm_flags &VM_READ ) { + action =MMU_MPROT_RONLY ; + }if (is_vm_hugetlb_page (vma)) - pages =hugetlb_change_protection (vma, start, end, newprot); + pages =hugetlb_change_protection (vma, start, end, newprot, action);else - pages =change_protection_range (vma, start, end, newprot, dirty_accountable, prot_numa); + pages =change_protection_range (vma, start, end, newprot, dirty_accountable, prot_numa, action);return pages; }
mm/mremap.c +2 -2
@@ -177,7 +177,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, mmun_start = old_addr; mmun_end = old_end; -mmu_notifier_invalidate_range_start (vma->vm_mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_start (vma->vm_mm, mmun_start, mmun_end,MMU_MREMAP );for (; old_addr < old_end; old_addr += extent, new_addr += extent) {cond_resched ();
@@ -221,7 +221,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,if (likely (need_flush))flush_tlb_range (vma, old_end-len, old_addr); -mmu_notifier_invalidate_range_end (vma->vm_mm, mmun_start, mmun_end); +mmu_notifier_invalidate_range_end (vma->vm_mm, mmun_start, mmun_end,MMU_MREMAP );return len + old_addr - old_end;/* how much done */ }
mm/rmap.c +49 -6
103 lines not on this page · raw message
virt/kvm/kvm_main.c +8 -4
@@ -262,7 +262,8 @@ static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)static void kvm_mmu_notifier_invalidate_page (struct mmu_notifier *mn,struct mm_struct *mm, -unsigned long address) +unsigned long address, +enum mmu_action action) {struct kvm *kvm =mmu_notifier_to_kvm (mn);int need_tlb_flush, idx;
@@ -301,7 +302,8 @@ static void kvm_mmu_notifier_invalidate_page(struct mmu_notifier *mn,static void kvm_mmu_notifier_change_pte (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long address, -pte_t pte) +pte_t pte, +enum mmu_action action) {struct kvm *kvm =mmu_notifier_to_kvm (mn);int idx;
@@ -317,7 +319,8 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,static void kvm_mmu_notifier_invalidate_range_start (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +enum mmu_action action) {struct kvm *kvm =mmu_notifier_to_kvm (mn);int need_tlb_flush =0 , idx;
@@ -343,7 +346,8 @@ static void kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,struct mm_struct *mm,unsigned long start, -unsigned long end) +unsigned long end, +enum mmu_action action) {struct kvm *kvm =mmu_notifier_to_kvm (mn);
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/11 mmu_notifier: pass through vma to invalidate_range and invalidate_page
In reply to [email protected]From: Jérôme Glisse <[email protected]>
New user of the mmu_notifier interface need to lookup vma in order to perform the invalidation operation. Instead of redoing a vma lookup inside the callback just pass through the vma from the call site where it is already available.
This needs small refactoring in memory.c to call invalidate_range on vma boundary the overhead should be low enough.
--- drivers/xen/gntdev.c | 4 +++- fs/proc/task_mmu.c | 17 ++++++++++++----- include/linux/mmu_notifier.h | 18 +++++++++++++++--- kernel/events/uprobes.c | 4 ++-- mm/filemap_xip.c | 2 +- mm/fremap.c | 4 ++-- mm/huge_memory.c | 26 +++++++++++++------------- mm/hugetlb.c | 16 ++++++++-------- mm/ksm.c | 8 ++++---- mm/memory.c | 38 ++++++++++++++++++++++++++------------ mm/migrate.c | 6 +++--- mm/mmu_notifier.c | 9 ++++++--- mm/mprotect.c | 4 ++-- mm/mremap.c | 4 ++-- mm/rmap.c | 8 ++++---- virt/kvm/kvm_main.c | 3 +++ 16 files changed, 106 insertions(+), 65 deletions(-)
drivers/xen/gntdev.c +3 -1
@@ -428,6 +428,7 @@ static void unmap_if_in_range(struct grant_map *map,static void mn_invl_range_start (struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action)
@@ -447,10 +448,11 @@ static void mn_invl_range_start(struct mmu_notifier *mn,static void mn_invl_page (struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long address,enum mmu_action action) { -mn_invl_range_start (mn, mm, address, address +PAGE_SIZE , action); +mn_invl_range_start (mn, mm, vma, address, address +PAGE_SIZE , action); } static void mn_release(struct mmu_notifier *mn,
fs/proc/task_mmu.c +12 -5
@@ -817,12 +817,19 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, .private = &cp, };down_read (&mm->mmap_sem); -if (type ==CLEAR_REFS_SOFT_DIRTY ) -mmu_notifier_invalidate_range_start (mm,0 ,-1 ,MMU_SOFT_DIRTY ); -for (vma = mm->mmap; vma; vma = vma->vm_next) +for (vma = mm->mmap; vma; vma = vma->vm_next) { +if (type ==CLEAR_REFS_SOFT_DIRTY ) +mmu_notifier_invalidate_range_start (mm, vma, + vma->vm_start, + vma->vm_end, +MMU_SOFT_DIRTY );walk_page_vma (vma, &clear_refs_walk); -if (type ==CLEAR_REFS_SOFT_DIRTY ) -mmu_notifier_invalidate_range_end (mm,0 ,-1 ,MMU_SOFT_DIRTY ); +if (type ==CLEAR_REFS_SOFT_DIRTY ) +mmu_notifier_invalidate_range_end (mm, vma, + vma->vm_start, + vma->vm_end, +MMU_SOFT_DIRTY ); + }flush_tlb_mm (mm);up_read (&mm->mmap_sem);mmput (mm);
include/linux/mmu_notifier.h +15 -3
@@ -126,6 +126,7 @@ struct mmu_notifier_ops {*/ void (*invalidate_page)(struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long address,enum mmu_action action);
@@ -174,11 +175,13 @@ struct mmu_notifier_ops {*/ void (*invalidate_range_start)(struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action);void (*invalidate_range_end)(struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action);
@@ -222,13 +225,16 @@ extern void __mmu_notifier_change_pte(struct mm_struct *mm,pte_t pte, enum mmu_action action);extern void __mmu_notifier_invalidate_page (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long address,enum mmu_action action);extern void __mmu_notifier_invalidate_range_start (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action);extern void __mmu_notifier_invalidate_range_end (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action);
@@ -265,29 +271,32 @@ static inline void mmu_notifier_change_pte(struct mm_struct *mm, }static inline void mmu_notifier_invalidate_page (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long address,enum mmu_action action) {if (mm_has_notifiers (mm)) -__mmu_notifier_invalidate_page (mm, address, action); +__mmu_notifier_invalidate_page (mm, vma, address, action); }static inline void mmu_notifier_invalidate_range_start (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action) {if (mm_has_notifiers (mm)) -__mmu_notifier_invalidate_range_start (mm, start, end, action); +__mmu_notifier_invalidate_range_start (mm, vma, start, end, action); }static inline void mmu_notifier_invalidate_range_end (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action) {if (mm_has_notifiers (mm)) -__mmu_notifier_invalidate_range_end (mm, start, end, action); +__mmu_notifier_invalidate_range_end (mm, vma, start, end, action); }static inline void mmu_notifier_mm_init (struct mm_struct *mm)
@@ -369,12 +378,14 @@ static inline void mmu_notifier_change_pte(struct mm_struct *mm, }static inline void mmu_notifier_invalidate_page (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long address,enum mmu_action action) { }static inline void mmu_notifier_invalidate_range_start (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action)
@@ -382,6 +393,7 @@ static inline void mmu_notifier_invalidate_range_start(struct mm_struct *mm, }static inline void mmu_notifier_invalidate_range_end (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action)
kernel/events/uprobes.c +2 -2
@@ -170,7 +170,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,/* For try_to_free_swap() and munlock_vma_page() below */ lock_page (page); -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_UNMAP ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_UNMAP ); err = -EAGAIN ; ptep =page_check_address (page, mm, addr, &ptl,0 );if (!ptep)
@@ -199,7 +199,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, err =0 ; unlock: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_UNMAP ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_UNMAP );unlock_page (page);return err; }
mm/filemap_xip.c +1 -1
@@ -198,7 +198,7 @@ retry:BUG_ON (pte_dirty (pteval));pte_unmap_unlock (pte, ptl);/* must invalidate_page _before_ freeing the page */ -mmu_notifier_invalidate_page (mm, address,MMU_UNMAP ); +mmu_notifier_invalidate_page (mm, vma, address,MMU_UNMAP );page_cache_release (page); } }
mm/fremap.c +2 -2
@@ -258,9 +258,9 @@ get_write_lock:* remapping is trickier as it can change the vma to non linear and thus * trigger side effect. */ -mmu_notifier_invalidate_range_start (mm, start, start + size,MMU_MUNMAP ); +mmu_notifier_invalidate_range_start (mm, vma, start, start + size,MMU_MUNMAP ); err = vma->vm_ops->remap_pages (vma, start, size, pgoff); -mmu_notifier_invalidate_range_end (mm, start, start + size,MMU_MUNMAP ); +mmu_notifier_invalidate_range_end (mm, vma, start, start + size,MMU_MUNMAP );/* * We can't clear VM_NONLINEAR because we'd have to do
mm/huge_memory.c +13 -13
@@ -993,7 +993,7 @@ static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm, mmun_start = haddr; mmun_end = haddr +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_THP_FAULT_WP ); ptl =pmd_lock (mm, pmd);if (unlikely (!pmd_same (*pmd, orig_pmd)))
@@ -1023,7 +1023,7 @@ static int do_huge_pmd_wp_page_fallback(struct mm_struct *mm,page_remove_rmap (page);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_FAULT_WP ); ret |=VM_FAULT_WRITE ;put_page (page);
@@ -1033,7 +1033,7 @@ out: out_free_pages:spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_FAULT_WP );mem_cgroup_uncharge_start ();for (i =0 ; i <HPAGE_PMD_NR ; i++) {mem_cgroup_uncharge_page (pages[i]);
@@ -1123,7 +1123,7 @@ alloc: mmun_start = haddr; mmun_end = haddr +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_THP_FAULT_WP );spin_lock (ptl);if (page)
@@ -1153,7 +1153,7 @@ alloc: }spin_unlock (ptl); out_mn: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_FAULT_WP ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_FAULT_WP ); out:return ret; out_unlock:
@@ -1588,7 +1588,7 @@ static int __split_huge_page_splitting(struct page *page,const unsigned long mmun_start = address;const unsigned long mmun_end = address +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT ); pmd =page_check_address_pmd (page, mm, address,PAGE_CHECK_ADDRESS_PMD_NOTSPLITTING_FLAG , &ptl);if (pmd) {
@@ -1603,7 +1603,7 @@ static int __split_huge_page_splitting(struct page *page, ret =1 ;spin_unlock (ptl); } -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT );return ret; }
@@ -2402,7 +2402,7 @@ static void collapse_huge_page(struct mm_struct *mm, mmun_start = address; mmun_end = address +HPAGE_PMD_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT ); pmd_ptl =pmd_lock (mm, pmd);/* probably unnecessary */ /* * After this gup_fast can't run anymore. This also removes
@@ -2412,7 +2412,7 @@ static void collapse_huge_page(struct mm_struct *mm,*/ _pmd =pmdp_clear_flush (vma, address, pmd);spin_unlock (pmd_ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT );spin_lock (pte_ptl); isolated =__collapse_huge_page_isolate (vma, address, pte);
@@ -2801,24 +2801,24 @@ void __split_huge_page_pmd(struct vm_area_struct *vma, unsigned long address, mmun_start = haddr; mmun_end = haddr +HPAGE_PMD_SIZE ; again: -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT ); ptl =pmd_lock (mm, pmd);if (unlikely (!pmd_trans_huge (*pmd))) {spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT );return ; }if (is_huge_zero_pmd (*pmd)) {__split_huge_zero_page_pmd (vma, haddr, pmd);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT );return ; } page =pmd_page (*pmd);VM_BUG_ON_PAGE (!page_count (page), page);get_page (page);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_THP_SPLIT ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_THP_SPLIT );split_huge_page (page);
mm/hugetlb.c +8 -8
@@ -2540,7 +2540,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, mmun_start = vma->vm_start; mmun_end = vma->vm_end;if (cow) -mmu_notifier_invalidate_range_start (src, mmun_start, mmun_end,MMU_COW ); +mmu_notifier_invalidate_range_start (src, vma, mmun_start, mmun_end,MMU_COW );for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) {spinlock_t *src_ptl, *dst_ptl;
@@ -2574,7 +2574,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, }if (cow) -mmu_notifier_invalidate_range_end (src, mmun_start, mmun_end,MMU_COW ); +mmu_notifier_invalidate_range_end (src, vma, mmun_start, mmun_end,MMU_COW );return ret; }
@@ -2626,7 +2626,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,BUG_ON (end & ~huge_page_mask (h));tlb_start_vma (tlb, vma); -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_UNMAP ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_UNMAP ); again:for (address = start; address < end; address += sz) { ptep =huge_pte_offset (mm, address);
@@ -2697,7 +2697,7 @@ unlock:if (address < end && !ref_page) goto again; } -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_UNMAP ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_UNMAP );tlb_end_vma (tlb, vma); }
@@ -2884,7 +2884,7 @@ retry_avoidcopy: mmun_start = address &huge_page_mask (h); mmun_end = mmun_start +huge_page_size (h); -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_UNMAP ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_UNMAP );/* * Retake the page table lock to check for racing updates * before the page tables are altered
@@ -2904,7 +2904,7 @@ retry_avoidcopy: new_page = old_page; }spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_UNMAP ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_UNMAP );page_cache_release (new_page);page_cache_release (old_page);
@@ -3342,7 +3342,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,BUG_ON (address >= end);flush_cache_range (vma, address, end); -mmu_notifier_invalidate_range_start (mm, start, end, action); +mmu_notifier_invalidate_range_start (mm, vma, start, end, action);mutex_lock (&vma->vm_file->f_mapping->i_mmap_mutex);for (; address < end; address +=huge_page_size (h)) {spinlock_t *ptl;
@@ -3372,7 +3372,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,*/ flush_tlb_range (vma, start, end);mutex_unlock (&vma->vm_file->f_mapping->i_mmap_mutex); -mmu_notifier_invalidate_range_end (mm, start, end, action); +mmu_notifier_invalidate_range_end (mm, vma, start, end, action);return pages << h->order; }
mm/ksm.c +4 -4
@@ -872,7 +872,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, mmun_start = addr; mmun_end = addr +PAGE_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_KSM_RONLY ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_KSM_RONLY ); ptep =page_check_address (page, mm, addr, &ptl,0 );if (!ptep)
@@ -912,7 +912,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, out_unlock:pte_unmap_unlock (ptep, ptl); out_mn: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_KSM_RONLY ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_KSM_RONLY ); out:return err; }
@@ -949,7 +949,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, mmun_start = addr; mmun_end = addr +PAGE_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_KSM ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_KSM ); ptep =pte_offset_map_lock (mm, pmd, addr, &ptl);if (!pte_same (*ptep, orig_pte)) {
@@ -972,7 +972,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,pte_unmap_unlock (ptep, ptl); err =0 ; out_mn: -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_KSM ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_KSM ); out:return err; }
mm/memory.c +26 -12
@@ -1054,7 +1054,7 @@ int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, mmun_start = addr; mmun_end = end;if (is_cow) -mmu_notifier_invalidate_range_start (src_mm, mmun_start, +mmu_notifier_invalidate_range_start (src_mm, vma, mmun_start, mmun_end,MMU_COW ); ret =0 ;
@@ -1072,7 +1072,7 @@ int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm, }while (dst_pgd++, src_pgd++, addr = next, addr != end);if (is_cow) -mmu_notifier_invalidate_range_end (src_mm, mmun_start, mmun_end, +mmu_notifier_invalidate_range_end (src_mm, vma, mmun_start, mmun_end,MMU_COW );return ret; }
@@ -1379,10 +1379,17 @@ void unmap_vmas(struct mmu_gather *tlb, {struct mm_struct *mm = vma->vm_mm; -mmu_notifier_invalidate_range_start (mm, start_addr, end_addr,MMU_MUNMAP ); -for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) +for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) { +mmu_notifier_invalidate_range_start (mm, vma, +max (start_addr, vma->vm_start), +min (end_addr, vma->vm_end), +MMU_MUNMAP );unmap_single_vma (tlb, vma, start_addr, end_addr,NULL ); -mmu_notifier_invalidate_range_end (mm, start_addr, end_addr,MMU_MUNMAP ); +mmu_notifier_invalidate_range_end (mm, vma, +max (start_addr, vma->vm_start), +min (end_addr, vma->vm_end), +MMU_MUNMAP ); + } }/**
@@ -1404,10 +1411,17 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start,lru_add_drain ();tlb_gather_mmu (&tlb, mm, start, end);update_hiwater_rss (mm); -mmu_notifier_invalidate_range_start (mm, start, end,MMU_MUNMAP ); -for ( ; vma && vma->vm_start < end; vma = vma->vm_next) +for ( ; vma && vma->vm_start < end; vma = vma->vm_next) { +mmu_notifier_invalidate_range_start (mm, vma, +max (start, vma->vm_start), +min (end, vma->vm_end), +MMU_MUNMAP );unmap_single_vma (&tlb, vma, start, end, details); -mmu_notifier_invalidate_range_end (mm, start, end,MMU_MUNMAP ); +mmu_notifier_invalidate_range_end (mm, vma, +max (start, vma->vm_start), +min (end, vma->vm_end), +MMU_MUNMAP ); + }tlb_finish_mmu (&tlb, start, end); }
@@ -1430,9 +1444,9 @@ static void zap_page_range_single(struct vm_area_struct *vma, unsigned long addrlru_add_drain ();tlb_gather_mmu (&tlb, mm, address, end);update_hiwater_rss (mm); -mmu_notifier_invalidate_range_start (mm, address, end,MMU_MUNMAP ); +mmu_notifier_invalidate_range_start (mm, vma, address, end,MMU_MUNMAP );unmap_single_vma (&tlb, vma, address, end, details); -mmu_notifier_invalidate_range_end (mm, address, end,MMU_MUNMAP ); +mmu_notifier_invalidate_range_end (mm, vma, address, end,MMU_MUNMAP );tlb_finish_mmu (&tlb, address, end); }
@@ -2851,7 +2865,7 @@ gotten: mmun_start = address &PAGE_MASK ; mmun_end = mmun_start +PAGE_SIZE ; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_FAULT_WP ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_FAULT_WP );/* * Re-check the pte - we dropped the lock
@@ -2920,7 +2934,7 @@ gotten: unlock:pte_unmap_unlock (page_table, ptl);if (mmun_end > mmun_start) -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_FAULT_WP ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_FAULT_WP );if (old_page) {/* * Don't let another task, with possibly unlocked vma,
mm/migrate.c +3 -3
@@ -1804,12 +1804,12 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,WARN_ON (PageLRU (new_page));/* Recheck the target PMD */ -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end,MMU_MIGRATE ); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end,MMU_MIGRATE ); ptl =pmd_lock (mm, pmd);if (unlikely (!pmd_same (*pmd, entry) ||page_count (page) !=2 )) { fail_putback:spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_MIGRATE ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_MIGRATE );/* Reverse changes made by migrate_page_copy() */ if (TestClearPageActive (new_page))
@@ -1875,7 +1875,7 @@ fail_putback:*/ mem_cgroup_end_migration (memcg, page, new_page, true);spin_unlock (ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end,MMU_MIGRATE ); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end,MMU_MIGRATE );/* Take an "isolate" reference and put new page on the LRU. */ get_page (new_page);
mm/mmu_notifier.c +6 -3
@@ -139,6 +139,7 @@ void __mmu_notifier_change_pte(struct mm_struct *mm, }void __mmu_notifier_invalidate_page (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long address,enum mmu_action action) {
@@ -148,12 +149,13 @@ void __mmu_notifier_invalidate_page(struct mm_struct *mm, id =srcu_read_lock (&srcu);hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) {if (mn->ops->invalidate_page) - mn->ops->invalidate_page (mn, mm, address, action); + mn->ops->invalidate_page (mn, mm, vma, address, action); }srcu_read_unlock (&srcu, id); }void __mmu_notifier_invalidate_range_start (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action)
@@ -165,13 +167,14 @@ void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, id =srcu_read_lock (&srcu);hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) {if (mn->ops->invalidate_range_start) - mn->ops->invalidate_range_start (mn, mm, start, end, action); + mn->ops->invalidate_range_start (mn, mm, vma, start, end, action); }srcu_read_unlock (&srcu, id); }EXPORT_SYMBOL_GPL (__mmu_notifier_invalidate_range_start);void __mmu_notifier_invalidate_range_end (struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action)
@@ -182,7 +185,7 @@ void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, id =srcu_read_lock (&srcu);hlist_for_each_entry_rcu (mn, &mm->mmu_notifier_mm->list, hlist) {if (mn->ops->invalidate_range_end) - mn->ops->invalidate_range_end (mn, mm, start, end, action); + mn->ops->invalidate_range_end (mn, mm, vma, start, end, action); }srcu_read_unlock (&srcu, id); }
mm/mprotect.c +2 -2
@@ -158,7 +158,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma,/* invoke the mmu notifier if the pmd is populated */ if (!mni_start) { mni_start = addr; -mmu_notifier_invalidate_range_start (mm, mni_start, end, action); +mmu_notifier_invalidate_range_start (mm, vma, mni_start, end, action); }if (pmd_trans_huge (*pmd)) {
@@ -186,7 +186,7 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, }while (pmd++, addr = next, addr != end);if (mni_start) -mmu_notifier_invalidate_range_end (mm, mni_start, end, action); +mmu_notifier_invalidate_range_end (mm, vma, mni_start, end, action);if (nr_huge_updates)count_vm_numa_events (NUMA_HUGE_PTE_UPDATES , nr_huge_updates);
mm/mremap.c +2 -2
@@ -177,7 +177,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, mmun_start = old_addr; mmun_end = old_end; -mmu_notifier_invalidate_range_start (vma->vm_mm, mmun_start, mmun_end,MMU_MREMAP ); +mmu_notifier_invalidate_range_start (vma->vm_mm, vma, mmun_start, mmun_end,MMU_MREMAP );for (; old_addr < old_end; old_addr += extent, new_addr += extent) {cond_resched ();
@@ -221,7 +221,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,if (likely (need_flush))flush_tlb_range (vma, old_end-len, old_addr); -mmu_notifier_invalidate_range_end (vma->vm_mm, mmun_start, mmun_end,MMU_MREMAP ); +mmu_notifier_invalidate_range_end (vma->vm_mm, vma, mmun_start, mmun_end,MMU_MREMAP );return len + old_addr - old_end;/* how much done */ }
mm/rmap.c +4 -4
@@ -834,7 +834,7 @@ static int page_mkclean_one(struct page *page, struct vm_area_struct *vma,pte_unmap_unlock (pte, ptl);if (ret) { -mmu_notifier_invalidate_page (mm, address,MMU_FILE_WB ); +mmu_notifier_invalidate_page (mm, vma, address,MMU_FILE_WB ); (*cleaned)++; } out:
@@ -1243,7 +1243,7 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma, out_unmap:pte_unmap_unlock (pte, ptl);if (ret !=SWAP_FAIL ) -mmu_notifier_invalidate_page (mm, address, action); +mmu_notifier_invalidate_page (mm, vma, address, action); out:return ret;
@@ -1347,7 +1347,7 @@ static int try_to_unmap_cluster(unsigned long cursor, unsigned int *mapcount, mmun_start = address; mmun_end = end; -mmu_notifier_invalidate_range_start (mm, mmun_start, mmun_end, action); +mmu_notifier_invalidate_range_start (mm, vma, mmun_start, mmun_end, action);/* * If we can acquire the mmap_sem for read, and vma is VM_LOCKED,
@@ -1412,7 +1412,7 @@ static int try_to_unmap_cluster(unsigned long cursor, unsigned int *mapcount, (*mapcount)--; }pte_unmap_unlock (pte -1 , ptl); -mmu_notifier_invalidate_range_end (mm, mmun_start, mmun_end, action); +mmu_notifier_invalidate_range_end (mm, vma, mmun_start, mmun_end, action);if (locked_vma)up_read (&vma->vm_mm->mmap_sem);return ret;
virt/kvm/kvm_main.c +3 -0
@@ -262,6 +262,7 @@ static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)static void kvm_mmu_notifier_invalidate_page (struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long address,enum mmu_action action) {
@@ -318,6 +319,7 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,static void kvm_mmu_notifier_invalidate_range_start (struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action)
@@ -345,6 +347,7 @@ static void kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,static void kvm_mmu_notifier_invalidate_range_end (struct mmu_notifier *mn,struct mm_struct *mm, +struct vm_area_struct *vma,unsigned long start,unsigned long end,enum mmu_action action)
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/11 interval_tree: helper to find previous item of a node in rb interval tree
In reply to [email protected]From: Jérôme Glisse <[email protected]>
It is often usefull to find the entry right before a given one in an rb interval tree.
--- include/linux/interval_tree_generic.h | 79 +++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+)
include/linux/interval_tree_generic.h +79 -0
@@ -188,4 +188,83 @@ ITPREFIX ## _iter_next(ITSTRUCT *node, ITTYPE start, ITTYPE last) \else if (start <=ITLAST (node))/* Cond2 */ \return node; \ } \ +} \ + \ +static ITSTRUCT * \ +ITPREFIX ## _subtree_rmost(ITSTRUCT *node, ITTYPE start, ITTYPE last) \ +{ \ +while (true) { \ +/* \ +* Loop invariant: last >= ITSTART(node) \ +* (Cond1 is satisfied) \ +*/ \ +if (node->ITRB.rb_right) { \ +ITSTRUCT *right =rb_entry (node->ITRB.rb_right, \ +ITSTRUCT ,ITRB ); \ +if (last >=ITSTART (right)) { \ +/* \ +* Some nodes in right subtree satisfy Cond1. \ +* Iterate to find the rightmost such node N. \ +* If it also satisfies Cond2, that's the \ +* match we are looking for. \ +*/ \ + node = right; \ +continue ; \ + } \ +/* Left branch might still have a candidate. */ \ +if (right->ITRB.rb_left) { \ + right =rb_entry (right->ITRB.rb_left, \ +ITSTRUCT ,ITRB ); \ +if (last >=ITSTART (right)) { \ + node = right; \ +continue ; \ + } \ + } \ + } \ +/* At this point node is the rightmost candidate. */ \ +if (last >=ITSTART (node)) {/* Cond1 */ \ +if (start <=ITLAST (node))/* Cond2 */ \ +return node;/* node is rightmost match */ \ + } \ +return NULL ;/* No match */ \ + } \ +} \ + \ +ITSTATIC ITSTRUCT * \ +ITPREFIX ## _iter_prev(ITSTRUCT *node, ITTYPE start, ITTYPE last) \ +{ \ +struct rb_node *rb = node->ITRB.rb_left, *prev; \ + \ +while (true) { \ +/* \ +* Loop invariants: \ +* Cond2: start <= ITLAST(node) \ +* rb == node->ITRB.rb_left \ +* \ +* First, search left subtree if suitable \ +*/ \ +if (rb) { \ +ITSTRUCT *left =rb_entry (rb,ITSTRUCT ,ITRB ); \ +if (start <= left->ITSUBTREE) \ + return ITPREFIX ## _subtree_rmost(left, \ + start, \ + last); \ + } \ + \ +/* Move up the tree until we come from a node's right child */ \ +do { \ + rb =rb_parent (&node->ITRB); \ +if (!rb) \ +return NULL ; \ + prev = &node->ITRB; \ + node =rb_entry (rb,ITSTRUCT ,ITRB ); \ + rb = node->ITRB.rb_left; \ + }while (prev == rb); \ + \ +/* Check if the node intersects [start;last] */ \ +if (start >ITLAST (node))/* !Cond2 */ \ +return NULL ; \ +else if (ITSTART (node) <= last)/* Cond1 */ \ +return node; \ + } \ }
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/11 mm/memcg: support accounting null page and transfering null charge to new page.
In reply to [email protected]From: Jérôme Glisse <[email protected]>
When migrating memory to some device specific memory we still want to properly account memcg memory usage. To do so we need to be able to account for page not allocated in system memory. We also need to be able to transfer previous charge from device memory to a page in an atomic way from memcg point of view.
Also introduce helper function to clear page memcg.
--- include/linux/memcontrol.h | 17 +++++ mm/memcontrol.c | 161 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 159 insertions(+), 19 deletions(-)
include/linux/memcontrol.h +17 -0
@@ -67,6 +67,8 @@ struct mem_cgroup_reclaim_cookie {extern int mem_cgroup_charge_anon (struct page *page,struct mm_struct *mm,gfp_t gfp_mask); +extern void mem_cgroup_transfer_charge_anon (struct page *page, +struct mm_struct *mm);/* for swap handling */ extern int mem_cgroup_try_charge_swapin (struct mm_struct *mm,struct page *page,gfp_t mask,struct mem_cgroup **memcgp);
@@ -85,6 +87,8 @@ extern void mem_cgroup_uncharge_start(void);extern void mem_cgroup_uncharge_end (void );extern void mem_cgroup_uncharge_page (struct page *page); +extern void mem_cgroup_uncharge_mm (struct mm_struct *mm); +extern void mem_cgroup_clear_page (struct page *page);extern void mem_cgroup_uncharge_cache_page (struct page *page); bool __mem_cgroup_same_or_subtree(const struct mem_cgroup *root_memcg,
@@ -245,6 +249,11 @@ static inline int mem_cgroup_charge_file(struct page *page,return 0 ; } +static inline void mem_cgroup_transfer_charge_anon (struct page *page, +struct mm_struct *mm) +{ +} +static inline int mem_cgroup_try_charge_swapin (struct mm_struct *mm,struct page *page,gfp_t gfp_mask,struct mem_cgroup **memcgp) {
@@ -272,6 +281,14 @@ static inline void mem_cgroup_uncharge_page(struct page *page) { } +static inline void mem_cgroup_uncharge_mm (struct mm_struct *mm) +{ +} + +static inline void mem_cgroup_clear_page (struct page *page) +{ +} +static inline void mem_cgroup_uncharge_cache_page (struct page *page) { }
mm/memcontrol.c +142 -19
213 lines not on this page · raw message
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/11 hmm: heterogeneous memory management
In reply to [email protected]From: Jérôme Glisse <[email protected]>
Motivation:
Heterogeneous memory management is intended to allow a device to transparently access a process address space without having to lock pages of the process or take references on them. In other word mirroring a process address space while allowing the regular memory management event such as page reclamation or page migration, to happen seamlessly.
Recent years have seen a surge into the number of specialized devices that are part of a computer platform (from desktop to phone). So far each of those devices have operated on there own private address space that is not link or expose to the process address space that is using them. This separation often leads to multiple memory copy happening between the device owned memory and the process memory. This of course is both a waste of cpu cycle and memory.
Over the last few years most of those devices have gained a full mmu allowing them to support multiple page table, page fault and other features that are found inside cpu mmu. There is now a strong incentive to start leveraging capabilities of such devices and to start sharing process address to avoid any unnecessary memory copy as well as simplifying the programming model of those devices by sharing an unique and common address space with the process that use them.
The aim of the heterogeneous memory management is to provide a common API that can be use by any such devices in order to mirror process address. The hmm code provide an unique entry point and interface itself with the core mm code of the linux kernel avoiding duplicate implementation and shielding device driver code from core mm code.
Moreover, hmm also intend to provide support for migrating memory to device private memory, allowing device to work on its own fast local memory. The hmm code would be responsible to intercept cpu page fault on migrated range of and to migrate it back to system memory allowing cpu to resume its access to the memory.
Another feature hmm intend to provide is support for atomic operation for the device even if the bus linking the device and the cpu do not have any such capabilities.
We expect that graphic processing unit and network interface to be among the first prominent users of such api.
Hardware requirement:
Because hmm is intended to be use by device driver there are minimum features requirement for the hardware mmu :
- hardware have its own page table per process (can be share btw != devices)
- hardware mmu support page fault and suspend execution until the page fault is serviced by hmm code. The page fault must also trigger some form of interrupt so that hmm code can be call by the device driver.
- hardware must support at least read only mapping (otherwise it can not access read only range of the process address space).
For better memory management it is highly recommanded that the device also support the following features :
- hardware mmu set access bit in its page table on memory access (like cpu).
- hardware page table can be updated from cpu or through a fast path.
- hardware provide advanced statistic over which range of memory it access the most.
- hardware differentiate atomic memory access from regular access allowing to support atomic operation even on platform that do not have atomic support with there bus link with the device.
Implementation:
The hmm layer provide a simple API to the device driver. Each device driver have to register and hmm device that holds pointer to all the callback the hmm code will make to synchronize the device page table with the cpu page table of a given process.
For each process it wants to mirror the device driver must register a mirror hmm structure that holds all the informations specific to the process being mirrored. Each hmm mirror uniquely link an hmm device with a process address space (the mm struct).
This design allow several different device driver to mirror concurrently the same process. The hmm layer will dispatch approprietly to each device driver modification that are happening to the process address space.
The hmm layer rely on the mmu notifier api to monitor change to the process address space. Because update to device page table can have unbound completion time, the hmm layer need the capability to sleep during mmu notifier callback.
This patch only implement the core of the hmm layer and do not support feature such as migration to device memory.
--- include/linux/hmm.h | 470 ++++++++++++++++++ include/linux/mm_types.h | 14 + kernel/fork.c | 6 + mm/Kconfig | 12 + mm/Makefile | 1 + mm/hmm.c | 1194 ++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 1697 insertions(+) create mode 100644 include/linux/hmm.h create mode 100644 mm/hmm.c
include/linux/hmm.h new file +470 -0
@@ -0,0 +1,470 @@ +/* +* Copyright 2013 Red Hat Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +* +* Authors: Jérôme Glisse <[email protected]> +*/ +/* This is a heterogeneous memory management (hmm). In a nutshell this provide +* an API to mirror a process address on a device which has its own mmu and its +* own page table for the process. It supports everything except special/mixed +* vma. +* +* To use this the hardware must have : +* - mmu with pagetable +* - pagetable must support read only (supporting dirtyness accounting is +* preferable but is not mandatory). +* - support pagefault ie hardware thread should stop on fault and resume +* once hmm has provided valid memory to use. +* - some way to report fault. +* +* The hmm code handle all the interfacing with the core kernel mm code and +* provide a simple API. It does support migrating system memory to device +* memory and handle migration back to system memory on cpu page fault. +* +* Migrated memory is considered as swaped from cpu and core mm code point of +* view. +*/ +#ifndef _HMM_H +#define _HMM_H + +#ifdef CONFIG_HMM + +#include <linux/list.h> +#include <linux/rwsem.h> +#include <linux/spinlock.h> +#include <linux/atomic.h> +#include <linux/mm_types.h> +#include <linux/mmu_notifier.h> +#include <linux/swap.h> +#include <linux/kref.h> +#include <linux/swapops.h> +#include <linux/mman.h> + + +struct hmm_device ; +struct hmm_device_ops ; +struct hmm_migrate ; +struct hmm_mirror ; +struct hmm_fault ; +struct hmm_event ; +struct hmm ; + +/* The hmm provide page informations to the device using hmm pfn value. Below +* are the various flags that define the current state the pfn is in (valid, +* type of page, dirty page, page is locked or not, ...). +* +* HMM_PFN_VALID_PAGE this means the pfn correspond to valid page. +* HMM_PFN_VALID_ZERO this means the pfn is the special zero page. +* HMM_PFN_DIRTY set when the page is dirty. +* HMM_PFN_WRITE is set if there is no need to call page_mkwrite +*/ +#define HMM_PFN_SHIFT (PAGE_SHIFT) +#define HMM_PFN_VALID_PAGE (0UL) +#define HMM_PFN_VALID_ZERO (1UL) +#define HMM_PFN_DIRTY (2UL) +#define HMM_PFN_WRITE (3UL) + +static inline struct page *hmm_pfn_to_page (unsigned long pfn) +{ +/* Ok to test on bit after the other as it can not flip from one to +* the other. Both bit are constant for the lifetime of an rmem +* object. +*/ +if (!test_bit (HMM_PFN_VALID_PAGE , &pfn) && + !test_bit (HMM_PFN_VALID_ZERO , &pfn)) { +return NULL ; + } +return pfn_to_page (pfn >>HMM_PFN_SHIFT ); +} + +static inline void hmm_pfn_set_dirty (unsigned long *pfn) +{ +set_bit (HMM_PFN_DIRTY , pfn); +} + + +/* hmm_fence - device driver fence to wait for device driver operations. +* +* In order to concurrently update several different devices mmu the hmm rely +* on device driver fence to wait for operation hmm has schedule to complete on +* the device. It is strongly recommanded to implement fences and have the hmm +* callback do as little as possible (just scheduling the update). Moreover the +* hmm code will reschedule for i/o the current process if necessary once it +* has scheduled all updates on all devices. +* +* Each fence is created as a result of either an update to range of memory or +* for remote memory to/from local memory dma. +* +* Update to range of memory correspond to a specific event type. For instance +* range of memory is unmap for page reclamation, or range of memory is unmap +* from process address as result of munmap syscall (HMM_RANGE_FINI), or there +* a memory protection change on the range. There is one hmm_etype for each of +* those event allowing the device driver to take appropriate action like for +* instance freeing device page table on HMM_RANGE_FINI but keeping it if it is +* HMM_RANGE_UNMAP (which means that the range is unmap but the range is still +* valid). +*/ +enum hmm_etype { +HMM_NONE =0 , +HMM_UNREGISTER , +HMM_DEVICE_FAULT , +HMM_MPROT_RONLY , +HMM_MPROT_RANDW , +HMM_MPROT_WONLY , +HMM_COW , +HMM_MUNMAP , +HMM_UNMAP , +HMM_MIGRATE_TO_LMEM , +HMM_MIGRATE_TO_RMEM , +}; + +struct hmm_fence { +struct list_head list; +struct hmm_mirror *mirror; +}; + + + + +/* hmm_device - Each device driver must register one and only one hmm_device. +* +* The hmm_device is the link btw hmm and each device driver. +*/ + +/* struct hmm_device_operations - hmm device operation callback +*/ +struct hmm_device_ops { +/* device_destroy - free hmm_device (call when refcount drop to 0). +* +* @device: The device hmm specific structure. +*/ +void (*device_destroy)(struct hmm_device *device); + +/* mirror_release() - device must stop using the address space. +* +* @mirror: The mirror that link process address space with the device. +* +* Called when as result of hmm_mirror_unregister or when mm is being +* destroy. +* +* It's illegal for the device to call any hmm helper function after +* this call back. The device driver must kill any pending device +* thread and wait for completion of all of them. +* +* Note that even after this callback returns the device driver might +* get call back from hmm. Callback will stop only once mirror_destroy +* is call. +*/ +void (*mirror_release)(struct hmm_mirror *hmm_mirror); + +/* mirror_destroy - free hmm_mirror (call when refcount drop to 0). +* +* @mirror: The mirror that link process address space with the device. +*/ +void (*mirror_destroy)(struct hmm_mirror *mirror); + +/* fence_wait() - to wait on device driver fence. +* +* @fence: The device driver fence struct. +* Returns: 0 on success,-EIO on error, -EAGAIN to wait again. +* +* Called when hmm want to wait for all operations associated with a +* fence to complete (including device cache flush if the event mandate +* it). +* +* Device driver must free fence and associated resources if it returns +* something else thant -EAGAIN. On -EAGAIN the fence must not be free +* as hmm will call back again. +* +* Return error if scheduled operation failed or if need to wait again. +* -EIO Some input/output error with the device. +* -EAGAIN The fence not yet signaled, hmm reschedule waiting thread. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +int (*fence_wait)(struct hmm_fence *fence); + +/* lmem_update() - update device mmu for a range of local memory. +* +* @mirror: The mirror that link process address space with the device. +* @faddr: First address in range (inclusive). +* @laddr: Last address in range (exclusive). +* @etype: The type of memory event (unmap, fini, read only, ...). +* @dirty: Device driver should call set_page_dirty_lock. +* Returns: Valid fence ptr or NULL on success otherwise ERR_PTR. +* +* Called to update device mmu permission/usage for a range of local +* memory. The event type provide the nature of the update : +* - range is no longer valid (munmap). +* - range protection changes (mprotect, COW, ...). +* - range is unmapped (swap, reclaim, page migration, ...). +* - ... +* +* Any event that block further write to the memory must also trigger a +* device cache flush and everything has to be flush to local memory by +* the time the wait callback return (if this callback returned a fence +* otherwise everything must be flush by the time the callback return). +* +* Device must properly call set_page_dirty on any page the device did +* write to since last call to update_lmem. This is only needed if the +* dirty parameter is true. +* +* The driver should return a fence pointer or NULL on success. It is +* advice to return fence and delay wait for the operation to complete +* to the wait callback. Returning a fence allow hmm to batch update to +* several devices and delay wait on those once they all have scheduled +* the update. +* +* Device driver must not fail lightly, any failure result in device +* process being kill. +* +* IMPORTANT IF DEVICE DRIVER GET HMM_MPROT_RANDW or HMM_MPROT_WONLY IT +* MUST NOT MAP SPECIAL ZERO PFN WITH WRITE PERMISSION. SPECIAL ZERO +* PFN IS SET THROUGH lmem_fault WITH THE HMM_PFN_VALID_ZERO BIT FLAG +* SET. +* +* Return fence or NULL on success, error value otherwise : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +struct hmm_fence *(*lmem_update)(struct hmm_mirror *mirror, +unsigned long faddr, +unsigned long laddr, +enum hmm_etype etype, +bool dirty); + +/* lmem_fault() - fault range of lmem on the device mmu. +* +* @mirror: The mirror that link process address space with the device. +* @faddr: First address in range (inclusive). +* @laddr: Last address in range (exclusive). +* @pfns: Array of pfn for the range (each of the pfn is valid). +* @fault: The fault structure provided by device driver. +* Returns: 0 on success, error value otherwise. +* +* Called to give the device driver each of the pfn backing a range of +* memory. It is only call as a result of a call to hmm_mirror_fault. +* +* Note that the pfns array content is only valid for the duration of +* the callback. Once the device driver callback return further memory +* activities might invalidate the value of the pfns array. The device +* driver will be inform of such changes through the update callback. +* +* Allowed return value are : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* Device driver must not fail lightly, any failure result in device +* process being kill. +* +* Return error if scheduled operation failed. Valid value : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +int (*lmem_fault)(struct hmm_mirror *mirror, +unsigned long faddr, +unsigned long laddr, +unsigned long *pfns, +struct hmm_fault *fault); +}; + +/* struct hmm_device - per device hmm structure +* +* @kref: Reference count. +* @mirrors: List of all active mirrors for the device. +* @mutex: Mutex protecting mirrors list. +* @ops: The hmm operations callback. +* @name: Device name (uniquely identify the device on the system). +* +* Each device that want to mirror an address space must register one of this +* struct (only once). +*/ +struct hmm_device { +struct kref kref; +struct list_head mirrors; +struct mutex mutex; +const struct hmm_device_ops *ops; +const char *name; +}; + +/* hmm_device_register() - register a device with hmm. +* +* @device: The hmm_device struct. +* @name: A unique name string for the device (use in error messages). +* Returns: 0 on success, -EINVAL otherwise. +* +* Call when device driver want to register itself with hmm. Device driver can +* only register once. It will return a reference on the device thus to release +* a device the driver must unreference the device. +*/ +int hmm_device_register (struct hmm_device *device,const char *name); + +struct hmm_device *hmm_device_ref (struct hmm_device *device); +struct hmm_device *hmm_device_unref (struct hmm_device *device); + + + + +/* hmm_mirror - device specific mirroring functions. +* +* Each device that mirror a process has a uniq hmm_mirror struct associating +* the process address space with the device. A process can be mirrored by +* several different devices at the same time. +*/ + +/* struct hmm_mirror - per device and per mm hmm structure +* +* @kref: Reference count. +* @dlist: List of all hmm_mirror for same device. +* @mlist: List of all hmm_mirror for same mm. +* @device: The hmm_device struct this hmm_mirror is associated to. +* @hmm: The hmm struct this hmm_mirror is associated to. +* @dead: The hmm_mirror is dead and should no longer be use. +* +* Each device that want to mirror an address space must register one of this +* struct for each of the address space it wants to mirror. Same device can +* mirror several different address space. As well same address space can be +* mirror by different devices. +*/ +struct hmm_mirror { +struct kref kref; +struct list_head dlist; +struct list_head mlist; +struct hmm_device *device; +struct hmm *hmm; +bool dead; +}; + +/* hmm_mirror_register() - register a device mirror against an mm struct +* +* @mirror: The mirror that link process address space with the device. +* @device: The device struct to associate this mirror with. +* @mm: The mm struct of the process. +* Returns: 0 success, -ENOMEM, -EBUSY or -EINVAL if process already mirrored. +* +* Call when device driver want to start mirroring a process address space. The +* hmm shim will register mmu_notifier and start monitoring process address +* space changes. Hence callback to device driver might happen even before this +* function return. +* +* The mm pin must also be hold (either task is current or using get_task_mm). +* +* Only one mirror per mm and hmm_device can be created, it will return -EINVAL +* if the hmm_device already has an hmm_mirror for the the mm. +* +* If the mm or previous hmm is in transient state then this will return -EBUSY +* and device driver must retry the call after unpinning the mm and checking +* again that the mm is valid. +* +* On success the mirror is returned with one reference for the caller, thus to +* release mirror call hmm_mirror_unref. +*/ +int hmm_mirror_register (struct hmm_mirror *mirror, +struct hmm_device *device, +struct mm_struct *mm); + +/* hmm_mirror_unregister() - unregister an hmm_mirror. +* +* @mirror: The mirror that link process address space with the device. +* +* Call when device driver want to stop mirroring a process address space. +*/ +void hmm_mirror_unregister (struct hmm_mirror *mirror); + +/* struct hmm_fault - device mirror fault informations +* +* @vma: The vma into which the fault range is (set by hmm). +* @faddr: First address of the range device want to fault (set by driver and +* updated by hmm to the actual first faulted address). +* @laddr: Last address of the range device want to fault (set by driver and +* updated by hmm to the actual last faulted address). +* @pfns: Array to hold the pfn value of each page in the range (provided by +* device driver, big enough to hold (laddr - faddr) >> PAGE_SHIFT). +* @flags: Fault flags (set by driver). +* +* This structure is given by the device driver to hmm_mirror_fault. The device +* driver can encapsulate the hmm_fault struct into its own fault structure and +* use that to provide private device driver information to the lmem_fault +* callback. +*/ +struct hmm_fault { +struct vm_area_struct *vma; +unsigned long faddr; +unsigned long laddr; +unsigned long *pfns; +unsigned long flags; +}; + +#define HMM_FAULT_WRITE (1 << 0) + +/* hmm_mirror_fault() - call by the device driver on device memory fault. +* +* @mirror: The mirror that link process address space with the device. +* @fault: The mirror fault struct holding fault range informations. +* +* Call when device is trying to access an invalid address in the device page +* table. The hmm shim will call lmem_fault with strong ordering in respect to +* call to lmem_update (ie any information provided to lmem_fault is valid +* until the device callback return). +* +* It will try to fault all pages in the range and give their pfn. If the vma +* covering the range needs to grow then it will. +* +* Also the fault will clamp the requested range to valid vma range (unless +* the vma into which event->faddr falls to, can grow). +* +* All error must be handled by device driver and most likely result in the +* process device tasks to be kill by the device driver. +* +* Returns: +* > 0 Number of pages faulted. +* -EINVAL if invalid argument. +* -ENOMEM if failing to allocate memory. +* -EACCES if trying to write to read only address (only for faddr). +* -EFAULT if trying to access an invalid address (only for faddr). +* -ENODEV if mirror is in process of being destroy. +*/ +int hmm_mirror_fault (struct hmm_mirror *mirror, +struct hmm_fault *fault); + +struct hmm_mirror *hmm_mirror_ref (struct hmm_mirror *mirror); +struct hmm_mirror *hmm_mirror_unref (struct hmm_mirror *mirror); + + + + +/* Functions used by core mm code. Device driver should not use any of them. */ +void __hmm_destroy (struct mm_struct *mm); +static inline void hmm_destroy (struct mm_struct *mm) +{ +if (mm->hmm) { +__hmm_destroy (mm); + } +} + +#else /* !CONFIG_HMM */ + +static inline void hmm_destroy (struct mm_struct *mm) +{ +} + +#endif /* !CONFIG_HMM */ + +#endif
include/linux/mm_types.h +14 -0
@@ -16,6 +16,10 @@#include <asm/page.h> #include <asm/mmu.h> +#ifdef CONFIG_HMM +struct hmm ; +#endif +#ifndef AT_VECTOR_SIZE_ARCH #define AT_VECTOR_SIZE_ARCH 0#endif
@@ -425,6 +429,16 @@ struct mm_struct {#ifdef CONFIG_MMU_NOTIFIER struct mmu_notifier_mm *mmu_notifier_mm;#endif +#ifdef CONFIG_HMM +/* +* hmm always register an mmu_notifier we rely on mmu notifier to keep +* refcount on mm struct as well as forbiding registering hmm on a +* dying mm +* +* This field is set with mmap_sem old in write mode. +*/ +struct hmm *hmm; +#endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS pgtable_t pmd_huge_pte;/* protected by page_table_lock */ #endif
kernel/fork.c +6 -0
@@ -27,6 +27,7 @@#include <linux/binfmts.h> #include <linux/mman.h> #include <linux/mmu_notifier.h> +#include <linux/hmm.h> #include <linux/fs.h> #include <linux/mm.h> #include <linux/vmacache.h>
@@ -602,6 +603,8 @@ void __mmdrop(struct mm_struct *mm)mm_free_pgd (mm);destroy_context (mm);mmu_notifier_mm_destroy (mm); +/* hmm_destroy needs to be call after mmu_notifier_mm_destroy */ +hmm_destroy (mm);check_mm (mm);free_mm (mm); }
@@ -820,6 +823,9 @@ static struct mm_struct *dup_mm(struct task_struct *tsk)memcpy (mm, oldmm,sizeof (*mm));mm_init_cpumask (mm); +#ifdef CONFIG_HMM + mm->hmm =NULL ; +#endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS mm->pmd_huge_pte =NULL ;
mm/Kconfig +12 -0
@@ -584,3 +584,15 @@ config PGTABLE_MAPPING config GENERIC_EARLY_IOREMAP bool + +config HMM + bool "Enable heterogeneous memory management (HMM)" + depends on MMU + select MMU_NOTIFIER + default n + help + Heterogeneous memory management provide infrastructure for a device + to mirror a process address space into an hardware mmu or into any + things supporting pagefault like event. + + If unsure, say N to disable hmm.
mm/Makefile +1 -0
@@ -63,3 +63,4 @@ obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o obj-$(CONFIG_ZBUD) += zbud.o obj-$(CONFIG_ZSMALLOC) += zsmalloc.o obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += early_ioremap.o +obj-$(CONFIG_HMM) += hmm.o
mm/hmm.c new file +1194 -0
1,194 lines not on this page · raw message
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/11 hmm: support moving anonymous page to remote memory
In reply to [email protected]From: Jérôme Glisse <[email protected]>
Motivation:
Migrating to device memory can allow device to access memory through a link with far greater bandwidth as well as with lower latency. Migration to device memory is of course only meaningfull if the memory will only be access by the device over a long period of time.
Because hmm aim to only provide an API to facilitate such use it does not deal with policy on when, what and to migrate to remote memory. It is expected that device driver that use hmm will have the informations to make such choice.
Implementation:
This use a two level structure to track remote memory. The first level is a range structure that match a range of address with a specific remote memory object. This allow for different range of address to point to the same remote memory object (usefull for shared memory).
The second level is a structure holding informations specific to hmm about the remote memory. This remote memory structure are allocated by device driver and thus can be included inside the remote memory structure that is specific to the device driver.
Each remote memory is given a range of unique id. Those unique id are use to create special hmm swap entry. For anonymous memory the cpu page table entry are set to this hmm swap entry and on cpu page fault the unique id is use to find the remote memory and migrate it back to system memory.
Other event than cpu page fault can trigger migration back to system memory. For instance on fork, to simplify things, the remote memory is migrated back to system memory.
--- include/linux/hmm.h | 469 ++++++++- include/linux/mmu_notifier.h | 1 + include/linux/swap.h | 12 +- include/linux/swapops.h | 33 +- mm/hmm.c | 2307 ++++++++++++++++++++++++++++++++++++++++-- mm/memcontrol.c | 46 + mm/memory.c | 7 + 7 files changed, 2768 insertions(+), 107 deletions(-)
include/linux/hmm.h +467 -2
@@ -56,10 +56,10 @@struct hmm_device ;struct hmm_device_ops ; -struct hmm_migrate ;struct hmm_mirror ;struct hmm_fault ;struct hmm_event ; +struct hmm_rmem ;struct hmm ;/* The hmm provide page informations to the device using hmm pfn value. Below
@@ -67,15 +67,34 @@ struct hmm;* type of page, dirty page, page is locked or not, ...). * * HMM_PFN_VALID_PAGE this means the pfn correspond to valid page. -* HMM_PFN_VALID_ZERO this means the pfn is the special zero page. +* HMM_PFN_VALID_ZERO this means the pfn is the special zero page either use +* it or directly clear rmem with zero what ever is the fastest method for +* the device. * HMM_PFN_DIRTY set when the page is dirty. * HMM_PFN_WRITE is set if there is no need to call page_mkwrite +* HMM_PFN_LOCK is only set while the rmem object is under going migration. +* HMM_PFN_LMEM_UPTODATE the page that is in the rmem pfn array has uptodate. +* HMM_PFN_RMEM_UPTODATE the rmem copy of the page is uptodate. +* +* Device driver only need to worry about : +* HMM_PFN_VALID_PAGE +* HMM_PFN_VALID_ZERO +* HMM_PFN_DIRTY +* HMM_PFN_WRITE +* Device driver must set/clear following flag after successfull dma : +* HMM_PFN_LMEM_UPTODATE +* HMM_PFN_RMEM_UPTODATE +* All the others flags are for hmm internal use only. */ #define HMM_PFN_SHIFT (PAGE_SHIFT) +#define HMM_PFN_CLEAR (((1UL << HMM_PFN_SHIFT) - 1UL) & ~0x3UL)#define HMM_PFN_VALID_PAGE (0UL)#define HMM_PFN_VALID_ZERO (1UL)#define HMM_PFN_DIRTY (2UL)#define HMM_PFN_WRITE (3UL) +#define HMM_PFN_LOCK (4UL) +#define HMM_PFN_LMEM_UPTODATE (5UL) +#define HMM_PFN_RMEM_UPTODATE (6UL)static inline struct page *hmm_pfn_to_page (unsigned long pfn) {
@@ -95,6 +114,28 @@ static inline void hmm_pfn_set_dirty(unsigned long *pfn)set_bit (HMM_PFN_DIRTY , pfn); } +static inline void hmm_pfn_set_lmem_uptodate (unsigned long *pfn) +{ +set_bit (HMM_PFN_LMEM_UPTODATE , pfn); +} + +static inline void hmm_pfn_set_rmem_uptodate (unsigned long *pfn) +{ +set_bit (HMM_PFN_RMEM_UPTODATE , pfn); +} + +static inline void hmm_pfn_clear_lmem_uptodate (unsigned long *pfn) +{ +clear_bit (HMM_PFN_LMEM_UPTODATE , pfn); +} + +static inline void hmm_pfn_clear_rmem_uptodate (unsigned long *pfn) +{ +clear_bit (HMM_PFN_RMEM_UPTODATE , pfn); +} + + +/* hmm_fence - device driver fence to wait for device driver operations. *
@@ -283,6 +324,255 @@ struct hmm_device_ops {unsigned long laddr, unsigned long *pfns, struct hmm_fault *fault); + +/* rmem_alloc - allocate a new rmem object. +* +* @device: Device into which to allocate the remote memory. +* @fault: The fault for which this remote memory is allocated. +* Returns: Valid rmem ptr on success, NULL or ERR_PTR otherwise. +* +* This allow migration to remote memory to operate in several steps. +* First the hmm code will clamp the range that can migrated and will +* unmap pages and prepare them for migration. +* +* It is only once migration is done with all above step that we know +* how much memory can be migrated which is when rmem_alloc is call to +* allocate the device rmem object to which memory should be migrated. +* +* Device driver can decide through this callback to abort migration +* by returning NULL, or it can decide to continue with migration by +* returning a properly allocated rmem object. +* +* Return rmem or NULL on success, error value otherwise : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +struct hmm_rmem *(*rmem_alloc)(struct hmm_device *device, +struct hmm_fault *fault); + +/* rmem_update() - update device mmu for a range of remote memory. +* +* @mirror: The mirror that link process address space with the device. +* @rmem: The remote memory under update. +* @faddr: First address in range (inclusive). +* @laddr: Last address in range (exclusive). +* @fuid: First uid of the remote memory at which the update begin. +* @etype: The type of memory event (unmap, fini, read only, ...). +* @dirty: Device driver should call hmm_pfn_set_dirty. +* Returns: Valid fence ptr or NULL on success otherwise ERR_PTR. +* +* Called to update device mmu permission/usage for a range of remote +* memory. The event type provide the nature of the update : +* - range is no longer valid (munmap). +* - range protection changes (mprotect, COW, ...). +* - range is unmapped (swap, reclaim, page migration, ...). +* - ... +* +* Any event that block further write to the memory must also trigger a +* device cache flush and everything has to be flush to remote memory by +* the time the wait callback return (if this callback returned a fence +* otherwise everything must be flush by the time the callback return). +* +* Device must properly call hmm_pfn_set_dirty on any page the device +* did write to since last call to update_rmem. This is only needed if +* the dirty parameter is true. +* +* The driver should return a fence pointer or NULL on success. It is +* advice to return fence and delay wait for the operation to complete +* to the wait callback. Returning a fence allow hmm to batch update to +* several devices and delay wait on those once they all have scheduled +* the update. +* +* Device driver must not fail lightly, any failure result in device +* process being kill. +* +* Return fence or NULL on success, error value otherwise : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +struct hmm_fence *(*rmem_update)(struct hmm_mirror *mirror, +struct hmm_rmem *rmem, +unsigned long faddr, +unsigned long laddr, +unsigned long fuid, +enum hmm_etype etype, +bool dirty); + +/* rmem_fault() - fault range of rmem on the device mmu. +* +* @mirror: The mirror that link process address space with the device. +* @rmem: The rmem backing this range. +* @faddr: First address in range (inclusive). +* @laddr: Last address in range (exclusive). +* @fuid: First rmem unique id (inclusive). +* @fault: The fault structure provided by device driver. +* Returns: 0 on success, error value otherwise. +* +* Called to give the device driver the remote memory that is backing a +* range of memory. The device driver can only map rmem page with write +* permission only if the HMM_PFN_WRITE bit is set. If device want to +* write to this range of rmem it can call hmm_mirror_fault. +* +* Return error if scheduled operation failed. Valid value : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +int (*rmem_fault)(struct hmm_mirror *mirror, +struct hmm_rmem *rmem, +unsigned long faddr, +unsigned long laddr, +unsigned long fuid, +struct hmm_fault *fault); + +/* rmem_to_lmem - copy remote memory to local memory. +* +* @rmem: The remote memory structure. +* @fuid: First rmem unique id (inclusive) of range to copy. +* @luid: Last rmem unique id (exclusive) of range to copy. +* Returns: Valid fence ptr or NULL on success otherwise ERR_PTR. +* +* This is call to copy remote memory back to local memory. The device +* driver need to schedule the dma to copy the remote memory to the +* pages given by the pfns array. Device driver should return a fence +* or an error pointer. +* +* If device driver does not return a fence then the device driver must +* wait until the dma is done and all device cache are flush. Moreover +* device driver must set the HMM_PFN_LMEM_UPTODATE on all successfully +* copied pages (setting this flag can be delayed to the fence_wait +* callback). +* +* If a valid fence is returned then hmm will wait on it and reschedule +* any thread that need rescheduling. +* +* DEVICE DRIVER MUST ABSOLUTELY TRY TO MAKE THIS CALL WORK OTHERWISE +* CPU THREAD WILL GET A SIGBUS. +* +* DEVICE DRIVER MUST SET HMM_PFN_LMEM_UPTODATE ON ALL SUCCESSFULLY +* COPIED PAGES. +* +* Return fence or NULL on success, error value otherwise : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +struct hmm_fence *(*rmem_to_lmem)(struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid); + +/* lmem_to_rmem - copy local memory to remote memory. +* +* @rmem: The remote memory structure. +* @fuid: First rmem unique id (inclusive) of range to copy. +* @luid: Last rmem unique id (exclusive) of range to copy. +* Returns: Valid fence ptr or NULL on success otherwise ERR_PTR. +* +* This is call to copy local memory to remote memory. The driver need +* to schedule the dma to copy the local memory from the pages given by +* the pfns array, to the remote memory. +* +* Device driver should return a fence or an error pointer. If device +* driver does not return a fence then the it must wait until the dma +* is done. The device driver must set the HMM_PFN_RMEM_UPTODATE on all +* successfully copied pages. +* +* If a valid fence is returned then hmm will wait on it and reschedule +* any thread that need rescheduling. +* +* Failure will result in aborting migration to remote memory. +* +* DEVICE DRIVER MUST SET HMM_PFN_RMEM_UPTODATE ON ALL SUCCESSFULLY +* COPIED PAGES. +* +* Return fence or NULL on success, error value otherwise : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +struct hmm_fence *(*lmem_to_rmem)(struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid); + +/* rmem_split - split rmem. +* +* @rmem: The remote memory to split. +* @fuid: First rmem unique id (inclusive) of range to split. +* @luid: Last rmem unique id (exclusive) of range to split. +* Returns: 0 on success, error value otherwise. +* +* Split remote memory, first the device driver must allocate a new +* remote memory struct, second it must call hmm_rmem_split_new and +* last it must transfer private driver resource from splited rmem to +* the new remote memory struct. +* +* Device driver _can not_ adjust nor the fuid nor the luid. +* +* Failure should be forwarded if any of the step fails. The device +* driver does not need to worry about freeing the new remote memory +* object once hmm_rmem_split_new is call as it will be freed through +* the rmem_destroy callback if anything fails. +* +* DEVICE DRIVER MUST ABSOLUTELY TRY TO MAKE THIS CALL WORK OTHERWISE +* THE WHOLE RMEM WILL BE MIGRATED BACK TO LMEM. +* +* Return error if operation failed. Valid value : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +int (*rmem_split)(struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid); + +/* rmem_split_adjust - split rmem. +* +* @rmem: The remote memory to split. +* @fuid: First rmem unique id (inclusive) of range to split. +* @luid: Last rmem unique id (exclusive) of range to split. +* Returns: 0 on success, error value otherwise. +* +* Split remote memory, first the device driver must allocate a new +* remote memory struct, second it must call hmm_rmem_split_new and +* last it must transfer private driver resource from splited rmem to +* the new remote memory struct. +* +* Device driver _can_ adjust the fuid or the luid with constraint that +* adjusted_fuid <= fuid and adjusted_luid >= luid. +* +* Failure should be forwarded if any of the step fails. The device +* driver does not need to worry about freeing the new remote memory +* object once hmm_rmem_split_new is call as it will be freed through +* the rmem_destroy callback if anything fails. +* +* DEVICE DRIVER MUST ABSOLUTELY TRY TO MAKE THIS CALL WORK OTHERWISE +* THE WHOLE RMEM WILL BE MIGRATED BACK TO LMEM. +* +* Return error if operation failed. Valid value : +* -ENOMEM Not enough memory for performing the operation. +* -EIO Some input/output error with the device. +* +* All other return value trigger warning and are transformed to -EIO. +*/ +int (*rmem_split_adjust)(struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid); + +/* rmem_destroy - destroy rmem. +* +* @rmem: The remote memory to destroy. +* +* Destroying remote memory structure once all ref are gone. +*/ +void (*rmem_destroy)(struct hmm_rmem *rmem); };/* struct hmm_device - per device hmm structure
@@ -292,6 +582,7 @@ struct hmm_device_ops {* @mutex: Mutex protecting mirrors list. * @ops: The hmm operations callback. * @name: Device name (uniquely identify the device on the system). +* @wait_queue: Wait queue for remote memory operations. * * Each device that want to mirror an address space must register one of this * struct (only once).
@@ -302,6 +593,8 @@ struct hmm_device {struct mutex mutex;const struct hmm_device_ops *ops;const char *name; +wait_queue_head_t wait_queue; +bool rmem; };/* hmm_device_register() - register a device with hmm.
@@ -322,6 +615,88 @@ struct hmm_device *hmm_device_unref(struct hmm_device *device); +/* hmm_rmem - The rmem struct hold hmm infos of a remote memory block. +* +* The device driver should derivate its remote memory tracking structure from +* the hmm_rmem structure. The hmm_rmem structure dos not hold any infos about +* the specific of the remote memory block (device address or anything else). +* It solely store informations needed for finding rmem when cpu try to access +* it. +*/ + +/* struct hmm_rmem - remote memory block +* +* @kref: Reference count. +* @device: The hmm device the remote memory is allocated on. +* @event: The event currently associated with the rmem. +* @lock: Lock protecting the ranges list and event field. +* @ranges: The list of address ranges that point to this rmem. +* @node: Node for rmem unique id tree. +* @pgoff: Page offset into file (in PAGE_SIZE not PAGE_CACHE_SIZE). +* @fuid: First unique id associated with this specific hmm_rmem. +* @fuid: Last unique id associated with this specific hmm_rmem. +* @subtree_luid: Optimization for red and black interval tree. +* @pfns: Array of pfn for local memory when some is attached. +* @dead: The remote memory is no longer valid restart lookup. +* +* Each hmm_rmem has a uniq range of id that is use to uniquely identify remote +* memory on cpu side. Those uniq id do not relate in any way with the device +* physical address at which the remote memory is located. +*/ +struct hmm_rmem { +struct kref kref; +struct hmm_device *device; +struct hmm_event *event; +spinlock_t lock; +struct list_head ranges; +struct rb_node node; +unsigned long pgoff; +unsigned long fuid; +unsigned long luid; +unsigned long subtree_luid; +unsigned long *pfns; +bool dead; +}; + +struct hmm_rmem *hmm_rmem_ref (struct hmm_rmem *rmem); +struct hmm_rmem *hmm_rmem_unref (struct hmm_rmem *rmem); + +/* hmm_rmem_split_new - helper to split rmem. +* +* @rmem: The remote memory to split. +* @new: The new remote memory struct. +* Returns: 0 on success, error value otherwise. +* +* The new remote memory struct must be allocated by the device driver and its +* fuid and lui field must be set to the range the device wish to new rmem to +* cover. +* +* Moreover all below conditions must be true : +* (new->fuid < new->luid) +* (new->fuid >= rmem->fuid && new->luid <= rmem->luid) +* (new->fuid == rmem->fuid || new->luid == rmem->luid) +* +* This hmm helper function will split range and perform internal hmm update on +* behalf of the device driver. +* +* Note that this function must be call by the rmem_split and rmem_split_adjust +* callback. +* +* Once this function is call the device driver should not try to free the new +* rmem structure no matter what is the return value. Moreover if the function +* return 0 then the device driver should properly update the new rmem struct. +* +* Return error if operation failed. Valid value : +* -EINVAL If one of the above condition is false. +* -ENOMEM If it failed to allocate memory. +* 0 on success. +*/ +int hmm_rmem_split_new (struct hmm_rmem *rmem, +struct hmm_rmem *new); + + + +/* hmm_mirror - device specific mirroring functions. * * Each device that mirror a process has a uniq hmm_mirror struct associating
@@ -406,6 +781,7 @@ void hmm_mirror_unregister(struct hmm_mirror *mirror);*/ struct hmm_fault {struct vm_area_struct *vma; +struct hmm_rmem *rmem;unsigned long faddr;unsigned long laddr;unsigned long *pfns;
@@ -450,6 +826,56 @@ struct hmm_mirror *hmm_mirror_unref(struct hmm_mirror *mirror); +/* hmm_migrate - Memory migration from local memory to remote memory. +* +* Below are functions that handle migration from local memory to remote memory +* (represented by hmm_rmem struct). This is a multi-step process first the +* range is unmap, then the device driver depending on the size of the unmaped +* range can decide to proceed or abort the migration. +*/ + +/* hmm_migrate_rmem_to_lmem() - force migration of some rmem to lmem. +* +* @mirror: The mirror that link process address space with the device. +* @faddr: First address of the range to migrate to lmem. +* @laddr: Last address of the range to migrate to lmem. +* Returns: 0 on success, -EIO or -EINVAL. +* +* This migrate any remote memory behind a range of address to local memory. +* +* Returns: +* 0 success. +* -EINVAL if invalid argument. +* -EIO if one of the device driver returned this error. +*/ +int hmm_migrate_rmem_to_lmem (struct hmm_mirror *mirror, +unsigned long faddr, +unsigned long laddr); + +/* hmm_migrate_lmem_to_rmem() - call to migrate lmem to rmem. +* +* @migrate: The migration temporary struct. +* @mirror: The mirror that link process address space with the device. +* Returns: 0, -EINVAL, -ENOMEM, -EFAULT, -EACCES, -ENODEV, -EBUSY, -EIO. +* +* On success the migrate struct is updated with the range that was migrated. +* +* Returns: +* 0 success. +* -EINVAL if invalid argument. +* -ENOMEM if failing to allocate memory. +* -EFAULT if range of address is invalid (no vma backing any of the range). +* -EACCES if vma backing the range is special vma. +* -ENODEV if mirror is in process of being destroy. +* -EBUSY if range can not be migrated (many different reasons). +* -EIO if one of the device driver returned this error. +*/ +int hmm_migrate_lmem_to_rmem (struct hmm_fault *fault, +struct hmm_mirror *mirror); + + + +/* Functions used by core mm code. Device driver should not use any of them. */ void __hmm_destroy (struct mm_struct *mm);static inline void hmm_destroy (struct mm_struct *mm)
@@ -459,12 +885,51 @@ static inline void hmm_destroy(struct mm_struct *mm) } } +/* hmm_mm_fault() - call when cpu pagefault on special hmm pte entry. +* +* @mm: The mm of the thread triggering the fault. +* @vma: The vma in which the fault happen. +* @addr: The address of the fault. +* @pte: Pointer to the pte entry inside the cpu page table. +* @pmd: Pointer to the pmd entry into which the pte is. +* @fault_flags: Fault flags (read, write, ...). +* @orig_pte: The original pte value when this fault happened. +* +* When the cpu try to access a range of memory that is in remote memory it +* fault in face of hmm special swap pte which will end up calling this +* function that should trigger the appropriate memory migration. +* +* Returns: +* 0 if some one else already migrated the rmem back. +* VM_FAULT_SIGBUS on any i/o error during migration. +* VM_FAULT_OOM if it fails to allocate memory for migration. +* VM_FAULT_MAJOR on successfull migration. +*/ +int hmm_mm_fault (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t *pte, +pmd_t *pmd, +unsigned int fault_flags, +pte_t orig_pte); +#else /* !CONFIG_HMM */ static inline void hmm_destroy (struct mm_struct *mm) { } +static inline int hmm_mm_fault (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t *pte, +pmd_t *pmd, +unsigned int fault_flags, +pte_t orig_pte) +{ +return VM_FAULT_SIGBUS ; +} +#endif /* !CONFIG_HMM */ #endif
include/linux/mmu_notifier.h +1 -0
@@ -42,6 +42,7 @@ enum mmu_action {MMU_FAULT_WP ,MMU_THP_SPLIT ,MMU_THP_FAULT_WP , + MMU_HMM, };#ifdef CONFIG_MMU_NOTIFIER
include/linux/swap.h +11 -1
@@ -70,8 +70,18 @@ static inline int current_is_kswapd(void)#define SWP_HWPOISON_NUM 0#endif +/* +* HMM (heterogeneous memory management) used when data is in remote memory. +*/ +#ifdef CONFIG_HMM +#define SWP_HMM_NUM 1 +#define SWP_HMM (MAX_SWAPFILES + SWP_MIGRATION_NUM + SWP_HWPOISON_NUM) +#else +#define SWP_HMM_NUM 0 +#endif +#define MAX_SWAPFILES \ - ((1 << MAX_SWAPFILES_SHIFT) - SWP_MIGRATION_NUM - SWP_HWPOISON_NUM) + ((1 << MAX_SWAPFILES_SHIFT) - SWP_MIGRATION_NUM - SWP_HWPOISON_NUM - SWP_HMM_NUM)/* * Magic header for a swap area. The first part of the union is
include/linux/swapops.h +32 -1
@@ -188,7 +188,38 @@ static inline int is_hwpoison_entry(swp_entry_t swp) }#endif -#if defined(CONFIG_MEMORY_FAILURE ) || defined(CONFIG_MIGRATION ) +#ifdef CONFIG_HMM + +static inline swp_entry_t make_hmm_entry (unsigned long pgoff) +{ +/* We don't need to keep the page pfn, so use offset to store writeable +* flag. +*/ +return swp_entry (SWP_HMM , pgoff); +} + +static inline unsigned long hmm_entry_uid (swp_entry_t entry) +{ +return swp_offset (entry); +} + +static inline int is_hmm_entry (swp_entry_t entry) +{ +return unlikely (swp_type (entry) ==SWP_HMM ); +} +#else /* !CONFIG_HMM */ +#define make_hmm_entry (page, write) swp_entry(0, 0) +static inline int is_hmm_entry (swp_entry_t swp) +{ +return 0 ; +} + +static inline void make_hmm_entry_read (swp_entry_t *entry) +{ +} +#endif /* !CONFIG_HMM */ + +#if defined(CONFIG_MEMORY_FAILURE ) || defined(CONFIG_MIGRATION ) || defined(CONFIG_HMM )static inline int non_swap_entry (swp_entry_t entry) {return swp_type (entry) >=MAX_SWAPFILES ;
mm/hmm.c +2204 -103
2,389 lines not on this page · raw message
mm/memcontrol.c +46 -0
46 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Part of the diff is not shown here. View the raw message.
PATCH 8/11 hmm: support for migrate file backed pages to remote memory
In reply to [email protected]From: Jérôme Glisse <[email protected]>
Motivation:
Same as for migrating anonymous private memory ie device local memory has higher bandwidth and lower latency.
Implementation:
Migrated range are tracked exactly as private anonymous memory refer to the commit adding support for migrating private anonymous memory.
Migrating file backed page is more complex than private anonymous memory as those pages might be involved in various filesystem event from write back to splice or truncation.
This patchset use a special hmm swap value that it store inside the radix tree for page that are migrated to remote memory. Any code that need to do radix tree lookup is updated to understand those special hmm swap entry and to call hmm helper function to perform the appropriate operation.
For most operations (file read, splice, truncate, ...) the end result is simply to migrate back to local memory. It is expected that user of hmm will do not perform such operation on file back memory that was migrated to remote memory.
Write back is different as we preserve the capabilities of doing dirtied memory write back from remote memory (using local system memory as a bounce buffer).
Each filesystem code must be modified to support hmm. This patchset only modify common helper code and add the core set of helpers needed for this feature.
Issues:
The big issue here is how to handle failure to migrate the remote memory back to local memory. Should all the process trying further access to the file get SIGBUS ? Should only the process that migrated memory to remote memory get SIGBUS ? ...
--- fs/aio.c | 9 + fs/buffer.c | 3 + fs/splice.c | 38 +- include/linux/fs.h | 4 + include/linux/hmm.h | 72 +++- include/linux/rmap.h | 1 + mm/filemap.c | 99 ++++- mm/hmm.c | 1094 ++++++++++++++++++++++++++++++++++++++++++++++++-- mm/madvise.c | 4 + mm/mincore.c | 11 + mm/page-writeback.c | 131 ++++-- mm/rmap.c | 17 +- mm/swap.c | 9 + mm/truncate.c | 103 ++++- 14 files changed, 1524 insertions(+), 71 deletions(-)
fs/aio.c +9 -0
@@ -40,6 +40,7 @@#include <linux/ramfs.h> #include <linux/percpu-refcount.h> #include <linux/mount.h> +#include <linux/hmm.h> #include <asm/kmap_types.h> #include <asm/uaccess.h>
@@ -405,10 +406,18 @@ static int aio_setup_ring(struct kioctx *ctx)for (i =0 ; i < nr_pages; i++) {struct page *page; + + repeat: page =find_or_create_page (file->f_inode->i_mapping, i,GFP_HIGHUSER | __GFP_ZERO);if (!page)break ; +if (radix_tree_exceptional_entry (page)) { +swp_entry_t swap =radix_to_swp_entry (page); + +hmm_pagecache_migrate (file->f_inode->i_mapping, swap); + goto repeat; + }pr_debug ("pid(%d) page[%d]->count=%d\n" , current->pid, i,page_count (page));SetPageUptodate (page);
fs/buffer.c +3 -0
@@ -40,6 +40,7 @@#include <linux/cpu.h> #include <linux/bitops.h> #include <linux/mpage.h> +#include <linux/hmm.h> #include <linux/bit_spinlock.h> #include <trace/events/block.h>
@@ -1023,6 +1024,8 @@ grow_dev_page(struct block_device *bdev, sector_t block,if (!page)return ret; +/* This can not happen ! */ +BUG_ON (radix_tree_exceptional_entry (page));BUG_ON (!PageLocked (page));if (page_has_buffers (page)) {
fs/splice.c +35 -3
@@ -33,6 +33,7 @@#include <linux/socket.h> #include <linux/compat.h> #include <linux/aio.h> +#include <linux/hmm.h> #include "internal.h" /*
@@ -334,6 +335,20 @@ __generic_file_splice_read(struct file *in, loff_t *ppos,* Lookup the (hopefully) full range of pages we need. */ spd.nr_pages =find_get_pages_contig (mapping, index, nr_pages, spd.pages); +/* Handle hmm entry, ie migrate remote memory back to local memory. */ +for (page_nr =0 ; page_nr < spd.nr_pages;) { + page = spd.pages[page_nr]; +if (radix_tree_exceptional_entry (page)) { +swp_entry_t swap =radix_to_swp_entry (page); + +/* FIXME How to handle hmm migration failure ? */ +hmm_pagecache_migrate (mapping, swap); + spd.pages[page_nr] =find_get_page (mapping, index + page_nr); +continue ; + }else { + page_nr++; + } + } index += spd.nr_pages;/*
@@ -351,6 +366,14 @@ __generic_file_splice_read(struct file *in, loff_t *ppos,* the first hole. */ page =find_get_page (mapping, index); + +if (radix_tree_exceptional_entry (page)) { +swp_entry_t swap =radix_to_swp_entry (page); + +/* FIXME How to handle hmm migration failure ? */ +hmm_pagecache_migrate (mapping, swap); +continue ; + }if (!page) {/* * page didn't exist, allocate one.
@@ -373,7 +396,6 @@ __generic_file_splice_read(struct file *in, loff_t *ppos,*/ unlock_page (page); } - spd.pages[spd.nr_pages++] = page; index++; }
@@ -415,6 +437,7 @@ __generic_file_splice_read(struct file *in, loff_t *ppos,*/ if (!page->mapping) {unlock_page (page); +retry: page =find_or_create_page (mapping, index,mapping_gfp_mask (mapping));
@@ -422,8 +445,17 @@ __generic_file_splice_read(struct file *in, loff_t *ppos, error = -ENOMEM ;break ; } -page_cache_release (spd.pages[page_nr]); - spd.pages[page_nr] = page; +/* At this point it can not be an exceptional hmm entry. */ +if (radix_tree_exceptional_entry (page)) { +swp_entry_t swap =radix_to_swp_entry (page); + +/* FIXME How to handle hmm migration failure ? */ +hmm_pagecache_migrate (mapping, swap); + goto retry; + }else { +page_cache_release (spd.pages[page_nr]); + spd.pages[page_nr] = page; + } }/* * page was already under io and is now done, great
include/linux/fs.h +4 -0
@@ -366,8 +366,12 @@ struct address_space_operations {int (*swap_activate)(struct swap_info_struct *sis,struct file *file,sector_t *span);void (*swap_deactivate)(struct file *file); + +int features; }; +#define AOPS_FEATURE_HMM (1 << 0) +extern const struct address_space_operations empty_aops;/*
include/linux/hmm.h +71 -1
120 lines not on this page · raw message
include/linux/rmap.h +1 -0
@@ -76,6 +76,7 @@ enum ttu_flags {TTU_POISON =1 ,/* unmap mode */ TTU_MIGRATION =2 ,/* migration mode */ TTU_MUNLOCK =3 ,/* munlock mode */ +TTU_HMM =4 ,/* hmm mode */ TTU_ACTION_MASK =0xff , TTU_IGNORE_MLOCK = (1 << 8), /* ignore mlock */
mm/filemap.c +94 -5
226 lines not on this page · raw message
mm/hmm.c +1069 -25
1,262 lines not on this page · raw message
mm/madvise.c +4 -0
@@ -202,6 +202,10 @@ static void force_shm_swapin_readahead(struct vm_area_struct *vma,continue ; } swap =radix_to_swp_entry (page); +if (is_hmm_entry (swap)) { +/* FIXME start migration here ? */ +continue ; + } page =read_swap_cache_async (swap,GFP_HIGHUSER_MOVABLE ,NULL ,0 );if (page)
mm/mincore.c +11 -0
@@ -79,6 +79,10 @@ static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff)*/ if (radix_tree_exceptional_entry (page)) {swp_entry_t swp =radix_to_swp_entry (page); + +if (is_hmm_entry (swp)) { +return 1 ; + } page =find_get_page (swap_address_space (swp), swp.val); } }else
@@ -86,6 +90,13 @@ static unsigned char mincore_page(struct address_space *mapping, pgoff_t pgoff)#else page =find_get_page (mapping, pgoff);#endif +if (radix_tree_exceptional_entry (page)) { +swp_entry_t swap =radix_to_swp_entry (page); + +if (is_hmm_entry (swap)) { +return 1 ; + } + }if (page) { present =PageUptodate (page);page_cache_release (page);
mm/page-writeback.c +98 -33
174 lines not on this page · raw message
mm/rmap.c +15 -2
@@ -1132,6 +1132,9 @@ int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,case TTU_MUNLOCK : action =MMU_MUNLOCK ;break ; +case TTU_HMM : + action =MMU_HMM ; +break ;default :/* Please report this ! */ BUG ();
@@ -1327,6 +1330,9 @@ static int try_to_unmap_cluster(unsigned long cursor, unsigned int *mapcount,case TTU_MUNLOCK : action =MMU_MUNLOCK ;break ; +case TTU_HMM : + action =MMU_HMM ; +break ;default :/* Please report this ! */ BUG ();
@@ -1426,7 +1432,12 @@ static int try_to_unmap_nonlinear(struct page *page,unsigned long cursor;unsigned long max_nl_cursor =0 ;unsigned long max_nl_size =0 ; -unsigned int mapcount; +unsigned int mapcount, min_mapcount =0 ; + +/* The hmm code keep mapcount elevated to 1 to avoid updating mm and +* memcg. If we are call on behalf of hmm just ignore this extra 1. +*/ + min_mapcount = (TTU_ACTION ((enum ttu_flags )arg) ==TTU_HMM ) ?1 :0 ;list_for_each_entry (vma, &mapping->i_mmap_nonlinear, shared.nonlinear) {
@@ -1449,8 +1460,10 @@ static int try_to_unmap_nonlinear(struct page *page,* just walk the nonlinear vmas trying to age and unmap some. * The mapcount of the page we came in with is irrelevant, * but even so use it as a guide to how hard we should try? +* +* See comment about hmm above for min_mapcount. */ - mapcount =page_mapcount (page); + mapcount =page_mapcount (page) - min_mapcount;if (!mapcount)return ret;
mm/swap.c +9 -0
@@ -839,6 +839,15 @@ void release_pages(struct page **pages, int nr, int cold)for (i =0 ; i < nr; i++) {struct page *page = pages[i]; +if (!page) { +continue ; + } +if (radix_tree_exceptional_entry (page)) { +/* This should really not happen tell us about it ! */ +WARN_ONCE (1 ,"hmm exceptional entry left\n" ); +continue ; + } +if (unlikely (PageCompound (page))) {if (zone) {spin_unlock_irqrestore (&zone->lru_lock, flags);
mm/truncate.c +101 -2
145 lines not on this page · raw message
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 9/11 fs/ext4: add support for hmm migration to remote memory of pagecache.
In reply to [email protected]From: Jérôme Glisse <[email protected]>
This add support for migrating page of ext4 filesystem to remote device memory using the hmm infrastructure. Writeback need special handling as we want to keep content inside remote memory.
--- fs/ext4/file.c | 20 +++++++ fs/ext4/inode.c | 175 +++++++++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 174 insertions(+), 21 deletions(-)
fs/ext4/file.c +20 -0
@@ -26,6 +26,7 @@#include <linux/aio.h> #include <linux/quotaops.h> #include <linux/pagevec.h> +#include <linux/hmm.h> #include "ext4.h" #include "ext4_jbd2.h" #include "xattr.h"
@@ -304,6 +305,7 @@ static int ext4_find_unwritten_pgoff(struct inode *inode,unsigned long nr_pages; num =min_t (pgoff_t, end - index,PAGEVEC_SIZE ); +retry: nr_pages =pagevec_lookup (&pvec, inode->i_mapping, index, (pgoff_t )num);if (nr_pages ==0 ) {
@@ -321,6 +323,24 @@ static int ext4_find_unwritten_pgoff(struct inode *inode,break ; } +for (i =0 ; i < nr_pages; i++) { +struct page *page = pvec.pages[i]; + +if (radix_tree_exception (page)) { +swp_entry_t swap =radix_to_swp_entry (page); + +/* FIXME How to handle hmm migration failure ? */ +hmm_pagecache_migrate (inode->i_mapping, swap); +for (; i < nr_pages; i++) { +if (radix_tree_exception (pvec.pages[i])) { + pvec.pages[i] =NULL ; + } + } +pagevec_release (&pvec); + goto retry; + } + } +/* * If this is the first time to go into the loop and * offset is smaller than the first page offset, it will be a
fs/ext4/inode.c +154 -21
261 lines not on this page · raw message
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/11 hmm/dummy: dummy driver to showcase the hmm api.
In reply to [email protected]From: Jérôme Glisse <[email protected]>
This is a dummy driver which full fill two purposes :
- showcase the hmm api and gives references on how to use it.
- provide an extensive user space api to stress test hmm.
This is a particularly dangerous module as it allow to access a mirror of a process address space through its device file. Hence it should not be enabled by default and only people actively developing for hmm should use it.
--- drivers/char/Kconfig | 9 + drivers/char/Makefile | 1 + drivers/char/hmm_dummy.c | 1128 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/hmm_dummy.h | 34 ++ 4 files changed, 1172 insertions(+) create mode 100644 drivers/char/hmm_dummy.c create mode 100644 include/uapi/linux/hmm_dummy.h
drivers/char/Kconfig +9 -0
@@ -600,5 +600,14 @@ config TILE_SROM device appear much like a simple EEPROM, and knows how to partition a single ROM for multiple purposes. +config HMM_DUMMY + tristate "hmm dummy driver to test hmm." + depends on HMM + default n + help + Say Y here if you want to build the hmm dummy driver that allow you + to test the hmm infrastructure by mapping a process address space + in hmm dummy driver device file. When in doubt, say "N". + endmenu
drivers/char/Makefile +1 -0
@@ -61,3 +61,4 @@ obj-$(CONFIG_JS_RTC) += js-rtc.o js-rtc-y = rtc.o obj-$(CONFIG_TILE_SROM) += tile-srom.o +obj-$(CONFIG_HMM_DUMMY) += hmm_dummy.o
drivers/char/hmm_dummy.c new file +1128 -0
1,128 lines not on this page · raw message
include/uapi/linux/hmm_dummy.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +* Copyright 2013 Red Hat Inc. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +* +* Authors: Jérôme Glisse <[email protected]> +*/ +/* This is a dummy driver made to exercice the HMM (hardware memory management) +* API of the kernel. It allow an userspace program to map its whole address +* space through the hmm dummy driver file. +*/ +#ifndef _UAPI_LINUX_HMM_DUMMY_H +#define _UAPI_LINUX_HMM_DUMMY_H + +#include <linux/types.h> +#include <linux/ioctl.h> +#include <linux/irqnr.h> + +/* Expose the address space of the calling process through hmm dummy dev file */ +#define HMM_DUMMY_EXPOSE_MM _IO( 'R', 0x00 ) + +#endif /* _UAPI_LINUX_RANDOM_H */
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 11/11 hmm/dummy_driver: add support for fake remote memory using pages.
In reply to [email protected]From: Jérôme Glisse <[email protected]>
Fake the existent of remote memory using preallocated pages and demonstrate how to use the hmm api related to remote memory.
--- drivers/char/hmm_dummy.c | 450 ++++++++++++++++++++++++++++++++++++++++- include/uapi/linux/hmm_dummy.h | 8 +- 2 files changed, 453 insertions(+), 5 deletions(-)
drivers/char/hmm_dummy.c +446 -4
@@ -48,6 +48,8 @@#define HMM_DUMMY_DEVICE_NAME "hmm_dummy_device"#define HMM_DUMMY_DEVICE_MAX_MIRRORS 4 +#define HMM_DUMMY_DEVICE_RMEM_SIZE (32UL << 20UL) +#define HMM_DUMMY_DEVICE_RMEM_NBITS (HMM_DUMMY_DEVICE_RMEM_SIZE >> PAGE_SHIFT)struct hmm_dummy_device ;
@@ -73,8 +75,16 @@ struct hmm_dummy_device {/* device file mapping tracking (keep track of all vma) */ struct hmm_dummy_mirror *dmirrors[HMM_DUMMY_DEVICE_MAX_MIRRORS ];struct address_space *fmapping[HMM_DUMMY_DEVICE_MAX_MIRRORS ]; +struct page **rmem_pages; +unsigned long *rmem_bitmap; }; +struct hmm_dummy_rmem { +struct hmm_rmem rmem; +unsigned long fuid; +unsigned long luid; +uint16_t *rmem_idx; +};/* We only create 2 device to show the inter device rmem sharing/migration * capabilities.
@@ -482,6 +492,51 @@ static void hmm_dummy_pt_free(struct hmm_dummy_mirror *dmirror, } +/* hmm_dummy_rmem - dummy remote memory using system memory pages +* +* Helper function to allocate fake remote memory out of the device rmem_pages. +*/ +static void hmm_dummy_rmem_free (struct hmm_dummy_rmem *drmem) +{ +struct hmm_dummy_device *ddevice; +struct hmm_rmem *rmem = &drmem->rmem; +unsigned long i, npages; + + npages = (rmem->luid - rmem->fuid); + ddevice = container_of(rmem->device, struct hmm_dummy_device, device); +mutex_lock (&ddevice->mutex); +for (i =0 ; i < npages; ++i) { +clear_bit (drmem->rmem_idx[i], ddevice->rmem_bitmap); + } +mutex_unlock (&ddevice->mutex); + +kfree (drmem->rmem_idx); + drmem->rmem_idx =NULL ; +} + +static struct hmm_dummy_rmem *hmm_dummy_rmem_new (void ) +{ +struct hmm_dummy_rmem *drmem; + + drmem =kzalloc (sizeof (*drmem),GFP_KERNEL ); +return drmem; +} + +static int hmm_dummy_mirror_lmem_to_rmem (struct hmm_dummy_mirror *dmirror, +unsigned long faddr, +unsigned long laddr) +{ +struct hmm_mirror *mirror = &dmirror->mirror; +struct hmm_fault fault; +int ret; + + fault.faddr = faddr &PAGE_MASK ; + fault.laddr =PAGE_ALIGN (laddr); + ret =hmm_migrate_lmem_to_rmem (&fault, mirror); +return ret; +} + +/* hmm_ops - hmm callback for the hmm dummy driver. * * Below are the various callback that the hmm api require for a device. The
@@ -574,7 +629,7 @@ static struct hmm_fence *hmm_dummy_lmem_update(struct hmm_mirror *mirror, page =hmm_dummy_pte_to_page (*pldp);if (page) { -set_page_dirty (page); +set_page_dirty_lock (page); } } *pldp &= ~HMM_DUMMY_PTE_DIRTY ;
@@ -631,6 +686,318 @@ static int hmm_dummy_lmem_fault(struct hmm_mirror *mirror,return 0 ; } +static struct hmm_rmem *hmm_dummy_rmem_alloc (struct hmm_device *device, +struct hmm_fault *fault) +{ +struct hmm_dummy_device *ddevice; +struct hmm_dummy_rmem *drmem; +struct hmm_rmem *rmem; +unsigned long i, npages; + + ddevice = container_of(device, struct hmm_dummy_device, device); + + drmem =hmm_dummy_rmem_new (); +if (drmem ==NULL ) { +return ERR_PTR (-ENOMEM ); + } + rmem = &drmem->rmem; + + npages = (fault->laddr - fault->faddr) >>PAGE_SHIFT ; + drmem->rmem_idx =kmalloc (npages *sizeof (uint16_t ),GFP_KERNEL ); +if (drmem->rmem_idx ==NULL ) { +kfree (drmem); +return ERR_PTR (-ENOMEM ); + } + +mutex_lock (&ddevice->mutex); +for (i =0 ; i < npages; ++i) { +int r; + + r =find_first_zero_bit (ddevice->rmem_bitmap, +HMM_DUMMY_DEVICE_RMEM_NBITS ); +if (r <0 ) { +while ((--i)) { +clear_bit (drmem->rmem_idx[i], + ddevice->rmem_bitmap); + } +kfree (drmem->rmem_idx); +kfree (drmem); +mutex_unlock (&ddevice->mutex); +return ERR_PTR (-ENOMEM ); + } + drmem->rmem_idx[i] = r; + } +mutex_unlock (&ddevice->mutex); + +return rmem; +} + +static struct hmm_fence *hmm_dummy_rmem_update (struct hmm_mirror *mirror, +struct hmm_rmem *rmem, +unsigned long faddr, +unsigned long laddr, +unsigned long fuid, +enum hmm_etype etype, +bool dirty) +{ +struct hmm_dummy_mirror *dmirror; +struct hmm_dummy_pt_map pt_map = {0 }; +unsigned long addr, i, mask, or, idx; + + dmirror = container_of(mirror, struct hmm_dummy_mirror, mirror); + pt_map.dmirror = dmirror; + idx = fuid - rmem->fuid; + +/* Sanity check for debugging hmm real device driver do not have to do that. */ +switch (etype) { +case HMM_UNREGISTER : +case HMM_UNMAP : +case HMM_MUNMAP : +case HMM_MPROT_WONLY : +case HMM_MIGRATE_TO_RMEM : +case HMM_MIGRATE_TO_LMEM : + mask =0 ; + or =0 ; +break ; +case HMM_MPROT_RONLY : +case HMM_WRITEBACK : + mask = ~HMM_DUMMY_PTE_WRITE ; + or =0 ; +break ; +case HMM_MPROT_RANDW : + mask =-1L ; + or =HMM_DUMMY_PTE_WRITE ; +break ; +default : +printk (KERN_ERR "%4d:%s invalid event type %d\n" , + __LINE__, __func__, etype); +return ERR_PTR (-EIO ); + } + +mutex_lock (&dmirror->mutex); +for (i =0 , addr = faddr; addr < laddr; ++i, addr +=PAGE_SIZE , ++idx) { +unsigned long *pldp; + + pldp =hmm_dummy_pt_pld_map (&pt_map, addr); +if (!pldp) { +continue ; + } +if (dirty && ((*pldp) &HMM_DUMMY_PTE_DIRTY )) { +hmm_pfn_set_dirty (&rmem->pfns[idx]); + } + *pldp &= ~HMM_DUMMY_PTE_DIRTY ; + *pldp &= mask; + *pldp |= or; + } +hmm_dummy_pt_unmap (&pt_map); + +switch (etype) { +case HMM_UNREGISTER : +case HMM_MUNMAP : +hmm_dummy_pt_free (dmirror, faddr, laddr); +break ; +default : +break ; + } +mutex_unlock (&dmirror->mutex); +return NULL ; +} + +static int hmm_dummy_rmem_fault (struct hmm_mirror *mirror, +struct hmm_rmem *rmem, +unsigned long faddr, +unsigned long laddr, +unsigned long fuid, +struct hmm_fault *fault) +{ +struct hmm_dummy_mirror *dmirror; +struct hmm_dummy_device *ddevice; +struct hmm_dummy_pt_map pt_map = {0 }; +struct hmm_dummy_rmem *drmem; +unsigned long i; +bool write = fault ? !!(fault->flags &HMM_FAULT_WRITE ) : false; + + dmirror = container_of(mirror, struct hmm_dummy_mirror, mirror); + drmem = container_of(rmem, struct hmm_dummy_rmem, rmem); + ddevice = dmirror->ddevice; + pt_map.dmirror = dmirror; + +mutex_lock (&dmirror->mutex); +for (i = fuid; faddr < laddr; ++i, faddr +=PAGE_SIZE ) { +unsigned long *pldp, pld_idx, pfn, idx = i - rmem->fuid; + + pldp =hmm_dummy_pt_pld_map (&pt_map, faddr); +if (!pldp) { +continue ; + } + pfn =page_to_pfn (ddevice->rmem_pages[drmem->rmem_idx[idx]]); + pld_idx =hmm_dummy_pld_index (faddr); + pldp[pld_idx] = (pfn <<HMM_DUMMY_PFN_SHIFT ); +if (test_bit (HMM_PFN_WRITE , &rmem->pfns[idx])) { + pldp[pld_idx] |=HMM_DUMMY_PTE_WRITE ; +hmm_pfn_clear_lmem_uptodate (&rmem->pfns[idx]); + } + pldp[pld_idx] |=HMM_DUMMY_PTE_VALID_PAGE ; +if (write && !test_bit (HMM_PFN_WRITE , &rmem->pfns[idx])) { +/* Fallback to use system memory. Other solution would be +* to migrate back to system memory. +*/ +hmm_pfn_clear_rmem_uptodate (&rmem->pfns[idx]); +if (!test_bit (HMM_PFN_LMEM_UPTODATE , &rmem->pfns[idx])) { +struct page *spage, *dpage; + + dpage =hmm_pfn_to_page (rmem->pfns[idx]); + spage = ddevice->rmem_pages[drmem->rmem_idx[idx]]; +copy_highpage (dpage, spage); +hmm_pfn_set_lmem_uptodate (&rmem->pfns[idx]); + } + pfn = rmem->pfns[idx] >>HMM_PFN_SHIFT ; + pldp[pld_idx] = (pfn <<HMM_DUMMY_PFN_SHIFT ); + pldp[pld_idx] |=HMM_DUMMY_PTE_WRITE ; + pldp[pld_idx] |=HMM_DUMMY_PTE_VALID_PAGE ; + } + } +hmm_dummy_pt_unmap (&pt_map); +mutex_unlock (&dmirror->mutex); +return 0 ; +} + +struct hmm_fence *hmm_dummy_rmem_to_lmem (struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid) +{ +struct hmm_dummy_device *ddevice; +struct hmm_dummy_rmem *drmem; +unsigned long i; + + ddevice = container_of(rmem->device, struct hmm_dummy_device, device); + drmem = container_of(rmem, struct hmm_dummy_rmem, rmem); + +for (i = fuid; i < luid; ++i) { +unsigned long idx = i - rmem->fuid; +struct page *spage, *dpage; + +if (test_bit (HMM_PFN_LMEM_UPTODATE , &rmem->pfns[idx])) { +/* This lmem page is already uptodate. */ +continue ; + } + spage = ddevice->rmem_pages[drmem->rmem_idx[idx]]; + dpage =hmm_pfn_to_page (rmem->pfns[idx]); +if (!dpage) { +return ERR_PTR (-EINVAL ); + } +copy_highpage (dpage, spage); +hmm_pfn_set_lmem_uptodate (&rmem->pfns[idx]); + } + +return NULL ; +} + +struct hmm_fence *hmm_dummy_lmem_to_rmem (struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid) +{ +struct hmm_dummy_device *ddevice; +struct hmm_dummy_rmem *drmem; +unsigned long i; + + ddevice = container_of(rmem->device, struct hmm_dummy_device, device); + drmem = container_of(rmem, struct hmm_dummy_rmem, rmem); + +for (i = fuid; i < luid; ++i) { +unsigned long idx = i - rmem->fuid; +struct page *spage, *dpage; + +if (test_bit (HMM_PFN_RMEM_UPTODATE , &rmem->pfns[idx])) { +/* This rmem page is already uptodate. */ +continue ; + } + dpage = ddevice->rmem_pages[drmem->rmem_idx[idx]]; + spage =hmm_pfn_to_page (rmem->pfns[idx]); +if (!spage) { +return ERR_PTR (-EINVAL ); + } +copy_highpage (dpage, spage); +hmm_pfn_set_rmem_uptodate (&rmem->pfns[idx]); + } + +return NULL ; +} + +static int hmm_dummy_rmem_do_split (struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid) +{ +struct hmm_dummy_rmem *drmem, *dnew; +struct hmm_fault fault; +struct hmm_rmem *new; +unsigned long i, pgoff, npages; +int ret; + + drmem = container_of(rmem, struct hmm_dummy_rmem, rmem); + npages = (luid - fuid); + pgoff = (fuid == rmem->fuid) ?0 : fuid - rmem->fuid; + fault.faddr =0 ; + fault.laddr = npages <<PAGE_SHIFT ; + new =hmm_dummy_rmem_alloc (rmem->device, &fault); +if (IS_ERR (new)) { +return PTR_ERR (new); + } + dnew = container_of(new, struct hmm_dummy_rmem, rmem); + + new->fuid = fuid; + new->luid = luid; + ret =hmm_rmem_split_new (rmem, new); +if (ret) { +return ret; + } + +/* Update the rmem it is fine to hold no lock as no one else can access +* both of this rmem object as long as the range are reserved. +*/ +for (i =0 ; i < npages; ++i) { + dnew->rmem_idx[i] = drmem->rmem_idx[i + pgoff]; + } +if (!pgoff) { +for (i =0 ; i < (rmem->luid - rmem->fuid); ++i) { + drmem->rmem_idx[i] = drmem->rmem_idx[i + npages]; + } + } + +return 0 ; +} + +static int hmm_dummy_rmem_split (struct hmm_rmem *rmem, +unsigned long fuid, +unsigned long luid) +{ +int ret; + +if (fuid > rmem->fuid) { + ret =hmm_dummy_rmem_do_split (rmem, rmem->fuid, fuid); +if (ret) { +return ret; + } + } +if (luid < rmem->luid) { + ret =hmm_dummy_rmem_do_split (rmem, luid, rmem->luid); +if (ret) { +return ret; + } + } + +return 0 ; +} + +static void hmm_dummy_rmem_destroy (struct hmm_rmem *rmem) +{ +struct hmm_dummy_rmem *drmem; + + drmem = container_of(rmem, struct hmm_dummy_rmem, rmem); +hmm_dummy_rmem_free (drmem); +kfree (drmem); +} +static const struct hmm_device_ops hmm_dummy_ops = { .device_destroy = &hmm_dummy_device_destroy, .mirror_release = &hmm_dummy_mirror_release,
@@ -638,6 +1005,14 @@ static const struct hmm_device_ops hmm_dummy_ops = { .fence_wait = &hmm_dummy_fence_wait, .lmem_update = &hmm_dummy_lmem_update, .lmem_fault = &hmm_dummy_lmem_fault, + .rmem_alloc = &hmm_dummy_rmem_alloc, + .rmem_update = &hmm_dummy_rmem_update, + .rmem_fault = &hmm_dummy_rmem_fault, + .rmem_to_lmem = &hmm_dummy_rmem_to_lmem, + .lmem_to_rmem = &hmm_dummy_lmem_to_rmem, + .rmem_split = &hmm_dummy_rmem_split, + .rmem_split_adjust = &hmm_dummy_rmem_split, + .rmem_destroy = &hmm_dummy_rmem_destroy, };
@@ -880,7 +1255,7 @@ static ssize_t hmm_dummy_fops_write(struct file *filp,if (!(pldp[pld_idx] &HMM_DUMMY_PTE_WRITE )) {hmm_dummy_pt_unmap (&pt_map);mutex_unlock (&dmirror->mutex); - goto fault; + goto fault; } pldp[pld_idx] |=HMM_DUMMY_PTE_DIRTY ; page =hmm_dummy_pte_to_page (pldp[pld_idx]);
@@ -964,8 +1339,11 @@ static long hmm_dummy_fops_unlocked_ioctl(struct file *filp, unsigned int command, unsigned long arg) { +struct hmm_dummy_migrate dmigrate;struct hmm_dummy_device *ddevice;struct hmm_dummy_mirror *dmirror; +struct hmm_mirror *mirror; +void __user *uarg = (void __user *)arg;unsigned minor;int ret;
@@ -1011,6 +1389,31 @@ static long hmm_dummy_fops_unlocked_ioctl(struct file *filp,"mirroring address space of %d\n" , dmirror->pid);return 0 ; +case HMM_DUMMY_MIGRATE_TO_RMEM : +mutex_lock (&ddevice->mutex); + dmirror = ddevice->dmirrors[minor]; +if (!dmirror) { +mutex_unlock (&ddevice->mutex); +return -EINVAL ; + } + mirror = &dmirror->mirror; +mutex_unlock (&ddevice->mutex); + +if (copy_from_user (&dmigrate, uarg,sizeof (dmigrate))) { +return -EFAULT ; + } + + ret =hmm_dummy_pt_alloc (dmirror, + dmigrate.faddr, + dmigrate.laddr); +if (ret) { +return ret; + } + + ret =hmm_dummy_mirror_lmem_to_rmem (dmirror, + dmigrate.faddr, + dmigrate.laddr); +return ret;default :return -EINVAL ; }
@@ -1034,7 +1437,31 @@ static const struct file_operations hmm_dummy_fops = {*/ static int hmm_dummy_device_init (struct hmm_dummy_device *ddevice) { -int ret, i; +struct page **pages; +unsigned long *bitmap; +int ret, i, npages; + + npages =HMM_DUMMY_DEVICE_RMEM_SIZE >>PAGE_SHIFT ; + bitmap =kzalloc (BITS_TO_LONGS (npages) *sizeof (long ),GFP_KERNEL ); +if (!bitmap) { +return -ENOMEM ; + } + pages =kzalloc (npages *sizeof (void *),GFP_KERNEL ); +if (!pages) { +kfree (bitmap); +return -ENOMEM ; + } +for (i =0 ; i < npages; ++i) { + pages[i] =alloc_page (GFP_KERNEL ); +if (!pages[i]) { +while ((--i)) { +__free_page (pages[i]); + } +kfree (bitmap); +kfree (pages); +return -ENOMEM ; + } + } ret = alloc_chrdev_region(&ddevice->dev, 0, HMM_DUMMY_DEVICE_MAX_MIRRORS,
@@ -1066,15 +1493,23 @@ static int hmm_dummy_device_init(struct hmm_dummy_device *ddevice) goto error; } + ddevice->rmem_bitmap = bitmap; + ddevice->rmem_pages = pages; +return 0 ; error: +for (i =0 ; i < npages; ++i) { +__free_page (pages[i]); + } +kfree (bitmap); +kfree (pages);return ret; }static void hmm_dummy_device_fini (struct hmm_dummy_device *ddevice) { -unsigned i; +unsigned i, npages;/* First finish hmm. */ for (i =0 ; i <HMM_DUMMY_DEVICE_MAX_MIRRORS ; i++) {
@@ -1092,6 +1527,13 @@ static void hmm_dummy_device_fini(struct hmm_dummy_device *ddevice)cdev_del (&ddevice->cdev);unregister_chrdev_region (ddevice->dev,HMM_DUMMY_DEVICE_MAX_MIRRORS ); + + npages =HMM_DUMMY_DEVICE_RMEM_SIZE >>PAGE_SHIFT ; +for (i =0 ; i < npages; ++i) { +__free_page (ddevice->rmem_pages[i]); + } +kfree (ddevice->rmem_bitmap); +kfree (ddevice->rmem_pages); }static int __init hmm_dummy_init (void )
include/uapi/linux/hmm_dummy.h +7 -1
@@ -29,6 +29,12 @@#include <linux/irqnr.h> /* Expose the address space of the calling process through hmm dummy dev file */ -#define HMM_DUMMY_EXPOSE_MM _IO( 'R', 0x00 ) +#define HMM_DUMMY_EXPOSE_MM _IO( 'R', 0x00 ) +#define HMM_DUMMY_MIGRATE_TO_RMEM _IO( 'R', 0x01 ) + +struct hmm_dummy_migrate { +uint64_t faddr; +uint64_t laddr; +};#endif /* _UAPI_LINUX_RANDOM_H */
Signature · 7 lines
1.9.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerSo you forgot to CC Linus, Linus has expressed some dislike for preemptible mmu_notifiers in the recent past:
https://lkml.org/lkml/2013/9/30/385
And here you're proposing to add dependencies on it.
Left the original msg in tact for the new Cc's.
Quoted 414 lines · On Fri, May 02, 2014 at 09:51:59AM -0400, [email protected] wrote:
In a nutshell:
The heterogeneous memory management (hmm) patchset implement a new api that sit on top of the mmu notifier api. It provides a simple api to device driver to mirror a process address space without having to lock or take reference on page and block them from being reclam or migrated. Any changes on a process address space is mirrored to the device page table by the hmm code. To achieve this not only we need each driver to implement a set of callback functions but hmm also interface itself in many key location of the mm code and fs code. Moreover hmm allow to migrate range of memory to the device remote memory to take advantages of its lower latency and higher bandwidth.
The why:
We want to be able to mirror a process address space so that compute api such as OpenCL or other similar api can start using the exact same address space on the GPU as on the CPU. This will greatly simplify usages of those api. Moreover we believe that we will see more and more specialize unit functions that will want to mirror process address using their own mmu.
To achieve this hmm requires :
A.1 - Hardware requirements
A.2 - sleeping insidemmu_notifier
A.3 - context information formmu_notifiercallback (patch 1 and 2) A.4 - new helper function for memcg (patch 5)
A.5 - special swap type and fault handling code
A.6 - file backed memory and filesystem changes
A.7 - The write back expectationWhile avoiding :
B.1 - No new page flag
B.2 - No special page reclamation codeFinally the rest of this email deals with :
C.1 - Alternative designs
C.2 - Hardware solution
C.3 - Routines markedEXPORT_SYMBOL
C.4 - Planned features
C.5 - Getting upstreamBut first patchlist :
0001 - Clarify the use of
TTU_UNMAPas being done for VMSCAN or POISONING 0002 - Give context information tommu_notifiercallback ie why the callback
is made for (because of munmap call, or page migration, ...).
0003 - Provide the vma for which the invalidation is happening tommu_notifier
callback. This is mostly and optimization to avoid looking up again the vma inside themmu_notifiercallback.
0004 - Add new helper to the generic interval tree (which use rb tree). 0005 - Add new helper to memcg so that anonymous page can be accounted as well
as unaccounted without a page struct. Also add a new helper function to transfer a charge to a page (charge which have been accounted without a struct page in the first place).
0006 - Introduce the hmm basic code to support simple device mirroring of the
address space. It is fully functional modulo some missing bit (guard or huge page and few other small corner cases).
0007 - Introduce support for migrating anonymous memory to device memory. This
involve introducing a new special swap type and teach the mm page fault code about hmm.
0008 - Introduce support for migrating shared or private memory that is backed
by a file. This is way more complex than anonymous case as it needs to synchronize with and exclude other kernel code path that might try to access those pages.
0009 - Add hmm support to ext4 filesystem.
0010 - Introduce a simple dummy driver that showcase use of the hmm api. 0011 - Add support for remote memory to the dummy driver.I believe that patch 1, 2, 3 are use full on their own as they could help fix some kvm issues (see https://lkml.org/lkml/2014/1/15/125) and they do not modify behavior of any current code (except that patch 3 might result in a larger number of call to
mmu_notifieras many as there is different vma for a range).Other patches have many rough edges but we would like to validate our design and see what we need to change before smoothing out any of them.
A.1 - Hardware requirements :
The hardware must have its own mmu with a page table per process it wants to mirror. The device mmu mandatory features are :
- per page read only flag.
- page fault support that stop/suspend hardware thread and support resuming those hardware thread once the page fault have been serviced.
- same number of bits for the virtual address as the target architecture (for instance 48 bits on current AMD 64).Advanced optional features :
- per page dirty bit (indicating the hardware did write to the page).
- per page access bit (indicating the hardware did access the page).A.2 - Sleeping in mmu notifier callback :
Because update device mmu might need to sleep, either for taking device driver lock (which might be consider fixable) or simply because invalidating the mmu might take several hundred millisecond and might involve allocating device or driver resources to perform the operation any of which might require to sleep.
Thus we need to be able to sleep inside
mmu_notifier_invalidate_range_startat the very least. Also we need to call tommu_notifier_change_pteto be bracketed bymmu_notifier_invalidate_range_startandmmu_notifier_invalidate_range_end. We need this becausemmu_notifier_change_pteis call with the anon vma lock held (and this is a non sleepable lock).A.3 - Context information for
mmu_notifiercallback :There is a need to provide more context information on why a
mmu_notifiercall back does happen. Was it because userspace call munmap ? Or was it because the kernel is trying to free some memory ? Or because page is being migrated ?The context is provided by using unique enum value associated with call site of
mmu_notifierfunctions. The patch here just add the enum value and modify each call site to pass along the proper value.The context information is important for management of the secondary mmu. For instance on a munmap the device driver will want to free all resources used by that range (device page table memory). This could as well solve the issue that was discussed in this thread https://lkml.org/lkml/2014/1/15/125 kvm can ignore
mmu_notifier_invalidate_rangebased on the enum value.A.4 - New helper function for memcg :
To keep memory control working as expect with the introduction of remote memory we need to add new helper function so we can account anonymous remote memory as if it was backed by a page. We also need to be able to transfer charge from the remote memory to pages and we need to be able clear a page cgroup without side effect to the memcg.
The patchset currently does add a new type of memory resource but instead just account remote memory as local memory (struct page) is. This is done with the minimum amount of change to the memcg code. I believe they are correct.
It might make sense to introduce a new sub-type of memory down the road so that device memory can be included inside the memcg accounting but we choose to not do so at first.
A.5 - Special swap type and fault handling code :
When some range of address is backed by device memory we need cpu fault to be aware of that so it can ask hmm to trigger migration back to local memory. To avoid too much code disruption we do so by adding a new special hmm swap type that is special cased in various place inside the mm page fault code. Refer to patch 7 for details.
A.6 - File backed memory and filesystem changes :
Using remote memory for range of address backed by a file is more complex than anonymous memory. There are lot more code path that might want to access pages that cache a file (for read, write, splice, ...). To avoid disrupting the code too much and sleeping inside page cache look up we decided to add hmm support on a per filesystem basis. So each filesystem can be teach about hmm and how to interact with it correctly.
The design is relatively simple, the radix tree is updated to use special hmm swap entry for any page which is in remote memory. Thus any radix tree look up will find the special entry and will know it needs to synchronize itself with hmm to access the file.
There is however subtleties. Updating the radix tree does not guarantee that hmm is the sole user of the page, another kernel/user thread might have done a radix look up before the radix tree update.
The solution to this issue is to first update the radix tree, then lock each page we are migrating, then unmap it from all the process using it and setting its mapping field to NULL so that once we unlock the page all existing code will thought that the page was either truncated or reclaimed in both cases all existing kernel code path will eith perform new look and see the hmm special entry or will just skip the page. Those code path were audited to insure that their behavior and expected result are not modified by this.
However this does not insure us exclusive access to the page. So at first when migrating such page to remote memory we map it read only inside the device and keep the page around so that both the device copy and the page copy contain the same data. If the device wishes to write to this remote memory then it call hmm fault code.
To allow write on remote memory hmm will try to free the page, if the page can be free then it means hmm is the unique user of the page and the remote memory can safely be written to. If not then this means that the page content might still be in use by some other process and the device driver have to choose to either wait or use the local memory instead. So local memory page are kept as long as there are other user for them. We likely need to hookup some special page reclamation code to force reclaiming those pages after a while.
A.7 - The write back expectation :
We also wanted to preserve the writeback and dirty balancing as we believe this is an important behavior (avoiding dirty content to stay for too long inside remote memory without being write back to disk). To avoid constantly migrating memory back and forth we decided to use existing page (hmm keep all shared page around and never free them for the lifetime of rmem object they are associated with) as temporary writeback source. On writeback the remote memory is mapped read only on the device and copied back to local memory which is use as source for the disk write.
This design choice can however be seen as counter productive as it means that the device using hmm will see its rmem map read only for writeback and then will have to wait for writeback to go through. Another choice would be to forget writeback while memory is on the device and pretend page are clear but this would break fsync and similar API for file that does have part of its content inside some device memory.
Middle ground might be to keep fsync and alike working but to ignore any other writeback.
B.1 - No new page flag :
While adding a new page flag would certainly help to find a different design to implement the hmm feature set. We tried to only think about design that do not require such a new flag.
B.2 - No special page reclamation code :
This is one of the big issue, should be isolate pages that are actively use by a device from the regular lru to a specific lru managed by the hmm code. In this patchset we decided to avoid doing so as it would just add complexity to already complex code.
Current code will trigger sleep inside vmscan when trying to reclaim page that belong to a process which is mirrored on a device. Is this acceptable or should we add a new hmm lru list that would handle all pages used by device in special way so that those pages are isolated from the regular page reclamation code.
C.1 - Alternative designs :
The current design is the one we believe provide enough ground to support all necessary features while keeping complexity as low as possible. However i think it is important to state that several others designs were tested and to explain why they were discarded.
D1) One of the first design introduced a secondary page table directly updated
by hmm helper functions. Hope was that this secondary page table could be in some way directly use by the device. That was naive ... to say the least.D2) The secondary page table with hmm specific format, was another design that
we tested. In this one the secondary page table was not intended to be use by the device but was intended to serve as a buffer btw the cpu page table and the device page table. Update to the device page table would use the hmm page table.While this secondary page table allow to track what is actively use and also gather statistics about it. It does require memory, in worst case as much as the cpu page table.
Another issue is that synchronization between cpu update and device trying to access this secondary page table was either prone to lock contention. Or was getting awfully complex to avoid locking all while duplicating complexity inside each of the device driver.
The killing bullet was however the fact that the code was littered with bug condition about discrepancy between the cpu and the hmm page table.
D3) Use a structure to track all actively mirrored range per process and per
device. This allow to have an exact view of which range of memory is in use by which device.Again this need a lot of memory to track each of the active range and worst case would need more memory than a secondary page table (one struct range per page).
Issue here was with the complexity or merging and splitting range on address space changes.
D4) Use a structure to track all active mirrored range per process (shared by
all the devices that mirror the same process). This partially address the memory requirement of D3 but this leave the complexity of range merging and splitting intact.The current design is a simplification of D4 in which we only track range of memory for memory that have been migrated to device memory. So for any others operations hmm directly access the cpu page table and forward the appropriate information to the device driver through the hmm api. We might need to go back to D4 design or a variation of it for some of the features we want add.
C.2 - Hardware solution :
What hmm try to achieve can be partially achieved using hardware solution. Such hardware solution is part of PCIE specification with the PASID (process address space id) and ATS (address translation service). With both of this PCIE feature a device can ask for a virtual address of a given process to be translated into its corresponding physical address. To achieve this the IOMMU bridge is capable of understanding and walking the cpu page table of a process. See the IOMMUv2 implementation inside the linux kernel for reference.
There is two huge restriction with hardware solution to this problem. First an obvious one is that you need hardware support. While HMM also require hardware support on the GPU side it does not on the architecture side (no requirement on IOMMU, or any bridges that are between the GPU and the system memory). This is a strong advantages to HMM it only require hardware support to one specific part.
The second restriction is that hardware solution like IOMMUv2 does not permit migrating chunk of memory to the device local memory which means under-using hardware resources (all discrete GPU comes with fast local memory that can have more than ten times the bandwidth of system memory).
This two reasons alone, are we believe enough to justify hmm usefulness.
Moreover hmm can work in a hybrid solution where non migrated chunk of memory goes through the hardware solution (IOMMUv2 for instance) and only the memory that is migrated to the device is handled by the hmm code. The requirement for the hardware is minimal, the hardware need to support the PASID & ATS (or any other hardware implementation of the same idea) on page granularity basis (it could be on the granularity of any level of the device page table so no need to populate all levels of the device page table). Which is the best solution for the problem.
C.3 - Routines marked
EXPORT_SYMBOLAs these routines are intended to be referenced in device drivers, they are marked
EXPORT_SYMBOLas is common practice. This encourages adoption of HMM in both GPL and non-GPL drivers, and allows ongoing collaboration with one of the primary authors of this idea.I think it would be beneficial to include this feature as soon as possible. Early collaborators can go to the trouble of fixing and polishing the HMM implementation, allowing it to fully bake by the time other drivers start implementing features requiring it. We are confident that this API will be useful to others as they catch up with supporting hardware.
C.4 - Planned features :
We are planning to add various features down the road once we can clear the basic design. Most important ones are :
- Allowing inter-device migration for compatible devices.
- Allowinghmm_rmemwithout backing storage (simplify some of the driver).
- Device specific memcg.
- Improvement to allow APU to take advantages of rmem, by hiding the page from the cpu the gpu can use a different memory controller link that do not require cache coherency with the cpu and thus provide higher bandwidth.
- Atomic device memory operation by unmapping on the cpu while the device is performing atomic operation (this require hardware mmu to differentiate between regular memory access and atomic memory access and to have a flag that allow atomic memory access on per page basis).
- Pining private memory to rmem this would be a useful feature to add and would require addition of a new flag to madvise. Any cpu access would result in SIGBUS for the cpu process.C.5 - Getting upstream :
So what should i do to get this patchset in a mergeable form at least at first as a staging feature ? Right now the patchset has few rough edges around huge page support and other smaller issues. But as said above i believe that patch 1, 2, 3 and 4 can be merge as is as they do not modify current behavior while being useful to other.
Should i implement some secondary hmm specific lru and their associated worker thread to avoid having the regular reclaim code to end up sleeping waiting for a device to update its page table ?
Should i go for a totaly different design ? If so what direction ? As stated above we explored other design and i listed there flaws.
Any others things that i need to fix/address/change/improve ?
Comments and flames are welcome.
Cheers,
Jérôme GlisseTo: <[email protected]>,
To: linux-mm <[email protected]>,
To: <[email protected]>,
Cc: "Mel Gorman" <[email protected]>,
Cc: "H. Peter Anvin" <[email protected]>,
Cc: "Peter Zijlstra" <[email protected]>,
Cc: "Andrew Morton" <[email protected]>,
Cc: "Linda Wang" <[email protected]>,
Cc: "Kevin E Martin" <[email protected]>,
Cc: "Jerome Glisse" <[email protected]>,
Cc: "Andrea Arcangeli" <[email protected]>,
Cc: "Johannes Weiner" <[email protected]>,
Cc: "Larry Woodman" <[email protected]>,
Cc: "Rik van Riel" <[email protected]>,
Cc: "Dave Airlie" <[email protected]>,
Cc: "Jeff Law" <[email protected]>,
Cc: "Brendan Conoboy" <[email protected]>,
Cc: "Joe Donohue" <[email protected]>,
Cc: "Duncan Poole" <[email protected]>,
Cc: "Sherry Cheung" <[email protected]>,
Cc: "Subhash Gutti" <[email protected]>,
Cc: "John Hubbard" <[email protected]>,
Cc: "Mark Hairgrove" <[email protected]>,
Cc: "Lucien Dunning" <[email protected]>,
Cc: "Cameron Buschardt" <[email protected]>,
Cc: "Arvind Gopalakrishnan" <[email protected]>,
Cc: "Haggai Eran" <[email protected]>,
Cc: "Or Gerlitz" <[email protected]>,
Cc: "Sagi Grimberg" <[email protected]>
Cc: "Shachar Raindel" <[email protected]>,
Cc: "Liran Liss" <[email protected]>,
Cc: "Roland Dreier" <[email protected]>,
Cc: "Sander, Ben" <[email protected]>,
Cc: "Stoner, Greg" <[email protected]>,
Cc: "Bridgman, John" <[email protected]>,
Cc: "Mantor, Michael" <[email protected]>,
Cc: "Blinzer, Paul" <[email protected]>,
Cc: "Morichetti, Laurent" <[email protected]>,
Cc: "Deucher, Alexander" <[email protected]>,
Cc: "Gabbay, Oded" <[email protected]>,
Quoted 3 lines · Peter Zijlstra
So you forgot to CC Linus, Linus has expressed some dislike for preemptible
mmu_notifiersin the recent past:
Indeed. I think we *really* should change that anonvma rwsem into an rwlock. We had performance numbers that showed it needs to be done.
The *last* thing we want is to have random callbacks that can block in this critical region. So now I think making it an rwlock is a good idea just to make sure that never happens.
Seriously, the mmu_notifiers were misdesigned to begin with, and much too deep. We're not screwing up the VM any more because of them.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Peter Zijlstra
So you forgot to CC Linus, Linus has expressed some dislike for preemptible
mmu_notifiersin the recent past:Indeed. I think we *really* should change that anonvma rwsem into an rwlock. We had performance numbers that showed it needs to be done.
Quoted 8 lines
The *last* thing we want is to have random callbacks that can block in this critical region. So now I think making it an rwlock is a good idea just to make sure that never happens.
Seriously, the
mmu_notifierswere misdesigned to begin with, and much too deep. We're not screwing up the VM any more because of them.Linus
So question becomes how to implement process address space mirroring without pinning memory and track cpu page table update knowing that device page table update is unbound can not be atomic from cpu point of view.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Jerome Glisse
So question becomes how to implement process address space mirroring without pinning memory and track cpu page table update knowing that device page table update is unbound can not be atomic from cpu point of view.
Perhaps as a fake TLB and interacting with the TLB shootdown? And making sure that everything is atomic?
Some of these devices are going to actually *share* the real page tables. Not "cache" them. Actually use the page tables directly. That's where all these on-die APU things are going, where the device really ends up being something much more like ASMP (asymmetric multi-processing) than a traditional external device.
So we *will* have to extend our notion of TLB shootdown to have not just a mask of possible active CPU's, but possible active devices. No question about that.
But doing this with sleeping in some stupid VM notifier is completely out of the question, because it *CANNOT EVEN WORK* for that eventual real goal of sharing the physical page tables where the device can do things like atomic dirty/accessed bit settings etc. It can only work for crappy sh*t that does the half-way thing. It's completely racy wrt the actual page table updates. That kind of page table sharing needs true atomicity for exactly the same reason we need it for our current SMP. So it needs to have all the same page table locking rules etc. Not that shitty notifier callback.
As I said, the VM notifiers were misdesigned to begin with. They are an abomination. We're not going to extend on that and make it worse. We are *certainly* not going to make them blocking and screwing our core VM that way. And that's doubly and triply true when it cannot work for the generic case _anyway_.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 16 lines · Linus Torvalds
Quoted 5 lines · Jerome Glisse
So question becomes how to implement process address space mirroring without pinning memory and track cpu page table update knowing that device page table update is unbound can not be atomic from cpu point of view.
Perhaps as a fake TLB and interacting with the TLB shootdown? And making sure that everything is atomic?
Some of these devices are going to actually *share* the real page tables. Not "cache" them. Actually use the page tables directly. That's where all these on-die APU things are going, where the device really ends up being something much more like ASMP (asymmetric multi-processing) than a traditional external device.
Quoted 3 lines
So we *will* have to extend our notion of TLB shootdown to have not just a mask of possible active CPU's, but possible active devices. No question about that.
Well no, as i said and explain in my mail APU and IOMMUv2 is a one sided coin and you can not use the device memory with such solution. So yes there is interest from many player to mirror the cpu page table by other means than by having the IOMMU walk the cpu page table (this include AMD).
Quoted 11 lines
But doing this with sleeping in some stupid VM notifier is completely out of the question, because it *CANNOT EVEN WORK* for that eventual real goal of sharing the physical page tables where the device can do things like atomic dirty/accessed bit settings etc. It can only work for crappy sh*t that does the half-way thing. It's completely racy wrt the actual page table updates. That kind of page table sharing needs true atomicity for exactly the same reason we need it for our current SMP. So it needs to have all the same page table locking rules etc. Not that shitty notifier callback.
Quoted 7 lines
As I said, the VM notifiers were misdesigned to begin with. They are an abomination. We're not going to extend on that and make it worse. We are *certainly* not going to make them blocking and screwing our core VM that way. And that's doubly and triply true when it cannot work for the generic case _anyway_.
Linus
So how can i solve the issue at hand. A device that has its own page table and can not mirror the cpu page table, nor can the device page table be updated atomicly from the cpu. Yes such device will exist and the IOMMUv2 walking the cpu page table is not capable of supporting GPU memory which is a big big big needed feature. Compare 20Gb/s vs 300Gb/s of GPU memory.
I understand that we do not want to sleep when updating process cpu page table but note that only process that use the gpu would have to sleep. So only process that can actually benefit from the using GPU will suffer the consequences.
That said it also play a role with page reclamation hence why i am proposing to have a separate lru for page involve with a GPU.
So having the hardware walking the cpu page table is out of the question.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Jerome Glisse
So how can i solve the issue at hand. A device that has its own page table and can not mirror the cpu page table, nor can the device page table be updated atomicly from the cpu. Yes such device will exist
TLB invalidation on very large systems can already take essentially forever.
Are we OK with extending that "forever" period for heterogeneous memory management with crappy devices, or is this something we could/should look into improving in the general case?
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Jerome Glisse
So how can i solve the issue at hand. A device that has its own page table and can not mirror the cpu page table, nor can the device page table be updated atomicly from the cpu.
So? Just model it as a TLB.
Sure, the TLB is slow and crappy and is in external memory rather than on-die, but it's still a TLB.
We have CPU's that do that kind of crazy thing (powerpc and sparc both have these kinds of "in-memory TLB extensions" in addition to the on-die TLB, they just call them "inverse page tables" to try to fool people about what they are).
Quoted 4 lines
I understand that we do not want to sleep when updating process cpu page table but note that only process that use the gpu would have to sleep. So only process that can actually benefit from the using GPU will suffer the consequences.
NO!
You don't get it. If a callback can sleep, then we cannot protect it with a spinlock.
It doesn't matter if it only sleeps once in a millennium. It still forces its crap on the rest of the system.
So there is no way in hell that we will allow that VM notifier crap. None.
And as I've mentioned, there is a correct place to slot this in, and that correct way is the _only_ way to ever support future GPU's that _do_ share direct access to the page tables.
So trying to do it any other way is broken _anyway_.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 21 lines · Linus Torvalds
Quoted 4 lines · Jerome Glisse
So how can i solve the issue at hand. A device that has its own page table and can not mirror the cpu page table, nor can the device page table be updated atomicly from the cpu.
So? Just model it as a TLB.
Sure, the TLB is slow and crappy and is in external memory rather than on-die, but it's still a TLB.
We have CPU's that do that kind of crazy thing (powerpc and sparc both have these kinds of "in-memory TLB extensions" in addition to the on-die TLB, they just call them "inverse page tables" to try to fool people about what they are).
Quoted 4 lines
I understand that we do not want to sleep when updating process cpu page table but note that only process that use the gpu would have to sleep. So only process that can actually benefit from the using GPU will suffer the consequences.
Quoted 7 lines
NO!
You don't get it. If a callback can sleep, then we cannot protect it with a spinlock.
It doesn't matter if it only sleeps once in a millennium. It still forces its crap on the rest of the system.
I do understand that i was pointing out that if i move to, tlb which i am fine with, i will still need to sleep there. That's all i wanted to stress, i did not wanted force using mmu_notifier, i am fine with them becoming atomic as long as i have a place where i can intercept cpu page table update and propagate them to device mmu.
Quoted 6 lines
So there is no way in hell that we will allow that VM notifier crap. None.
And as I've mentioned, there is a correct place to slot this in, and that correct way is the _only_ way to ever support future GPU's that _do_ share direct access to the page tables.
This work was done in cooperation with NVidia and we discussed with AMD too so i am very much aware of what is coming next on hardware front and being able to have GPU have their own GPU page table ie not walking the CPU one is something of interest to the people who design those future generation of GPU.
Quoted 4 lines
So trying to do it any other way is broken _anyway_.
Linus
I will respin without using mmu_notifier and by hooking it as tlb shootdown. But it will still need to sleep during the device tlb shootdown and that's the point i want to make sure is clear.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Jerome Glisse
I do understand that i was pointing out that if i move to, tlb which i am fine with, i will still need to sleep there.
No can do. The TLB flushing itself is called with a spinlock held, and we need to continue to do that.
Why do you really need to sleep? Because that sounds bogus.
What you *should* do is send the flush message, and not wait for any reply. You can then possibly wait for the result later on: we already have this multi-stage TLB flush model (using the "mmu_gather" structure) that has three phases:
- create mmu_gather (allocate space for batching etc). This can sleep.
- do the actual flushing (possibly multiple times). This is the
"synchronous with the VM" part and cannot sleep.
- tear down the mmu_gather data structures and actually free the
pages we batched. This can sleep.
and what I think a GPU flush has to do is to do the actual flushes when asked to (because that's what it will need to do to work with a real TLB eventually), but if there's some crazy asynchronous acknowledge thing from hardware, it's possible to perhaps wait for that in the final phase (*before* we free the pages we gathered).
Now, such an asynchronous model had better not mark page tables dirty after we flushed (we'd lose that information), but quite frankly, anything that is remote enough to need some async flush thing cannor sanely be close enough to be closely tied to the actual real page tables, so I don't think we need to care.
Anyway, I really think that the existing mmu_gather model *should* work fine for this all. It may be a bit inconvenient for crazy hardware, but the important part is that it definitely should work for any future hardware that actually gets this right.
It does likely involve adding some kind of device list to "struct mm_struct", and I'm sure there's some extension to "struct mmu_gather" too, but _conceptually_ it should all be reasonably non-invasive.
Knock wood.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 21 lines · Linus Torvalds
Quoted 3 lines · Jerome Glisse
I do understand that i was pointing out that if i move to, tlb which i am fine with, i will still need to sleep there.
No can do. The TLB flushing itself is called with a spinlock held, and we need to continue to do that.
Why do you really need to sleep? Because that sounds bogus.
What you *should* do is send the flush message, and not wait for any reply. You can then possibly wait for the result later on: we already have this multi-stage TLB flush model (using the "
mmu_gather" structure) that has three phases:- create
mmu_gather(allocate space for batching etc). This can sleep.
- do the actual flushing (possibly multiple times). This is the
"synchronous with the VM" part and cannot sleep.
- tear down themmu_gatherdata structures and actually free the
pages we batched. This can sleep.
Quoted 5 lines
and what I think a GPU flush has to do is to do the actual flushes when asked to (because that's what it will need to do to work with a real TLB eventually), but if there's some crazy asynchronous acknowledge thing from hardware, it's possible to perhaps wait for that in the final phase (*before* we free the pages we gathered).
Plan i had in mind was to add an item atomicly inside mmu notifier to schedule work on the gpu and have the tlb wait on the gpu to acknowledge that it did update its page table and it is done using those pages. This would happen in tlb_flush_mmu
Quoted 6 lines
Now, such an asynchronous model had better not mark page tables dirty after we flushed (we'd lose that information), but quite frankly, anything that is remote enough to need some async flush thing cannor sanely be close enough to be closely tied to the actual real page tables, so I don't think we need to care.
That's an issue as soon as i schedule the work (read as early on as i can) the gpu can report any of the page as dirty and it can possibly do so only once we wait for it in tlb_flush_mmu.
Quoted 6 lines
Anyway, I really think that the existing
mmu_gathermodel *should* work fine for this all. It may be a bit inconvenient for crazy hardware, but the important part is that it definitely should work for any future hardware that actually gets this right.
I stress again the GPU with dedicated memory is not going away on the opposite you might see more dedicated memory not accessible from the CPU.
Quoted 7 lines
It does likely involve adding some kind of device list to "struct
mm_struct", and I'm sure there's some extension to "structmmu_gather" too, but _conceptually_ it should all be reasonably non-invasive.Knock wood.
Linus
I will port over to piggy back on mmu gather and other tlb flush. I will post as soon as i have something that works with features this patchset has.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 38 lines · Linus Torvalds
Quoted 3 lines · Jerome Glisse
I do understand that i was pointing out that if i move to, tlb which i am fine with, i will still need to sleep there.
No can do. The TLB flushing itself is called with a spinlock held, and we need to continue to do that.
Why do you really need to sleep? Because that sounds bogus.
What you *should* do is send the flush message, and not wait for any reply. You can then possibly wait for the result later on: we already have this multi-stage TLB flush model (using the "
mmu_gather" structure) that has three phases:- create
mmu_gather(allocate space for batching etc). This can sleep.
- do the actual flushing (possibly multiple times). This is the
"synchronous with the VM" part and cannot sleep.
- tear down themmu_gatherdata structures and actually free the
pages we batched. This can sleep.and what I think a GPU flush has to do is to do the actual flushes when asked to (because that's what it will need to do to work with a real TLB eventually), but if there's some crazy asynchronous acknowledge thing from hardware, it's possible to perhaps wait for that in the final phase (*before* we free the pages we gathered).
Now, such an asynchronous model had better not mark page tables dirty after we flushed (we'd lose that information), but quite frankly, anything that is remote enough to need some async flush thing cannor sanely be close enough to be closely tied to the actual real page tables, so I don't think we need to care.
Anyway, I really think that the existing
mmu_gathermodel *should* work fine for this all. It may be a bit inconvenient for crazy hardware, but the important part is that it definitely should work for any future hardware that actually gets this right.
Quoted 7 lines
It does likely involve adding some kind of device list to "struct
mm_struct", and I'm sure there's some extension to "structmmu_gather" too, but _conceptually_ it should all be reasonably non-invasive.Knock wood.
Linus
Also, just to be sure, are my changes to the radix tree otherwise acceptable at least in principle. As explained in my long email when migrating file backed page to device memory we want to make sure that no one else try to use those pages.
The way i have done it is described in my long email but in a nutshell it use special swap entry inside the radix tree and have filesystem code knows about that and call into the hmm to migrate back to memory if needed. Writeback use a temporary bounce page (ie once writeback is done the gpu page can be remapped writeable and the bounce page forgotten).
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Jerome Glisse
Also, just to be sure, are my changes to the radix tree otherwise acceptable at least in principle.
It looks like it adds several "loop over each page" cases just to check whether each page might be on remote memory.
Which seems a complete waste of time 99.99% of the time.
But maybe I'm looking at the wrong patch.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Jerome Glisse
Also, just to be sure, are my changes to the radix tree otherwise acceptable at least in principle.
Quoted 8 lines
It looks like it adds several "loop over each page" cases just to check whether each page might be on remote memory.
Which seems a complete waste of time 99.99% of the time.
But maybe I'm looking at the wrong patch.
Linus
It is patch 8 for core changes and patch 9 to demonstrate per fs changes.
So yes each place that does radix tree lookup need to check that the entries it got out of the radix tree are not special one and because many place in the code gang lookup with pagevec_lookup there is need to go over entries that were looked up to take appropriate action.
Other design is to do migration inside the various radix tree lookup functions but that means going out of rcu section and possibly sleeping waiting for the GPU to copy back things into system memory.
I could grow the radix function to return some bool to avoid looping over for case where there is no special entry.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Jerome Glisse
I could grow the radix function to return some bool to avoid looping over for case where there is no special entry.
.. or even just a bool (or counter) associated with the mapping to mark whether any special entries exist at all.
Also, the code to turn special entries is duplicated over and over again, usually together with a "FIXME - what about migration failure", so it would make sense to do that as it's own function.
But conceptually I don't hate it. I didn't much like having random hmm_pagecache_migrate() calls in core vm code, and code like this
+ hmm_pagecache_migrate(mapping, swap);
+ spd.pages[page_nr] = find_get_page(mapping,
index + page_nr);
looks fundamentally racy, and in other places you seemed to assume that all exceptional entries are always about hmm, which looked questionable. But those are details. The concept of putting a special swap entry in the mapping radix trees I don't necessarily find objectionable per se.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Jerome Glisse
I could grow the radix function to return some bool to avoid looping over for case where there is no special entry.
Quoted 7 lines
.. or even just a bool (or counter) associated with the mapping to mark whether any special entries exist at all.
Also, the code to turn special entries is duplicated over and over again, usually together with a "FIXME - what about migration failure", so it would make sense to do that as it's own function.
Migration failure is when something goes horribly wrong and GPU can not copy back the page to system memory that philosophical question associated is what to do about other process ? Make them SIGBUS ?
The answer so far is consider this as any kind of cpu thread that would crash and only half write content it wanted into the page. So other thread will use the lastest version of the data we have. Thread that triggered the migration to the GPU memory would see a SIGBUS (those thread are GPU aware as they use some form of GPU api such as OpenCL).
Quoted 7 lines
But conceptually I don't hate it. I didn't much like having random
hmm_pagecache_migrate()calls in core vm code, and code like this(quoted diff) +2
-0+ hmm_pagecache_migrate(mapping, swap); + spd.pages[page_nr] = find_get_page(mapping,index +
page_nr);
Quoted 7 lines
looks fundamentally racy, and in other places you seemed to assume that all exceptional entries are always about hmm, which looked questionable. But those are details. The concept of putting a special swap entry in the mapping radix trees I don't necessarily find objectionable per se.
Linus
So far only shmem use special entry and my patchset did not support it as i wanted to vet the design first.
The hmm_pagecache_migrate is the function that trigger migration back to system memory. Once again the expectation is that such code path will neve be call, only the process that use the GPU and the mmaped file will ever access those pages and this process knows that it should not access them while they are on the GPU so if it does it has to suffer the consequences.
Thanks a lot for all the feedback, much appreciated.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Linus Torvalds
and what I think a GPU flush has to do is to do the actual flushes when asked to (because that's what it will need to do to work with a real TLB eventually), but if there's some crazy asynchronous acknowledge thing from hardware, it's possible to perhaps wait for that in the final phase (*before* we free the pages we gathered).
Hrm, difficult. We have some pretty strong assumptions that ptep_clear_flush() is fully synchronous as far as I can tell... ie, your trick would work for the unmap case but everything else is still problematic.
Ben.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Jerome Glisse
I do understand that i was pointing out that if i move to, tlb which i am fine with, i will still need to sleep there. That's all i wanted to stress, i did not wanted force using
mmu_notifier, i am fine with them becoming atomic as long as i have a place where i can intercept cpu page table update and propagate them to device mmu.
Your MMU notifier can maintain a map of "dirty" PTEs and you do the actual synchronization in the subsequent flush_tlb_* , you need to add hooks there but it's much less painful than in the notifiers.
*However* Linus, even then we can't sleep. We do things like ptep_clear_flush() that need the PTL and have the synchronous flush semantics.
Sure, today we wait, possibly for a long time, with IPIs, but we do not sleep. Jerome would have to operate within a similar context. No sleep for you :)
Cheers,
Ben.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 6 lines · Jerome Glisse
I do understand that i was pointing out that if i move to, tlb which i am fine with, i will still need to sleep there. That's all i wanted to stress, i did not wanted force using
mmu_notifier, i am fine with them becoming atomic as long as i have a place where i can intercept cpu page table update and propagate them to device mmu.Your MMU notifier can maintain a map of "dirty" PTEs and you do the actual synchronization in the subsequent flush_tlb_* , you need to add hooks there but it's much less painful than in the notifiers.
Well getting back the dirty info from the GPU also require to sleep. Maybe i should explain how it is suppose to work. GPU have several command buffer and execute instructions inside those command buffer in sequential order. To update the GPU mmu you need to schedule command into one of those command buffer but when you do so you do not know how much command are in front of you and how long it will take to the GPU to get to your command.
Yes GPU this patchset target have preemption but it is not as flexible as CPU preemption there is not kernel thread running and scheduling, all the scheduling is done in hardware. So the preemption is more limited that on CPU.
That is why any update or information retrieval from the GPU need to go through some command buffer and no matter how high priority the command buffer for mmu update is, it can still long time (think flushing thousand of GPU thread and saving there context).
Quoted 11 lines
*However* Linus, even then we can't sleep. We do things like
ptep_clear_flush()that need the PTL and have the synchronous flush semantics.Sure, today we wait, possibly for a long time, with IPIs, but we do not sleep. Jerome would have to operate within a similar context. No sleep for you :)
Cheers,
Ben.
So for the ptep_clear_flush my idea is to have a special lru for page that are in use by the GPU. This will prevent the page reclaimation try_to_unmap and thus the ptep_clear_flush. I would block ksm so again another user that would no do ptep_clear_flush. I would need to fix remap_file_pages either adding some callback there or refactor the unmap and tlb flushing.
Finaly for page migration i see several solutions, forbid it (easy for me but likely not what we want) have special code inside migrate code to handle page in use by a device, or have special code inside try_to_unmap to handle it.
I think this is all the current user of ptep_clear_flush and derivative that does flush tlb while holding spinlock.
Note that for special lru or event special handling of page in use by a device i need a new page flag. Would this be acceptable ?
For the special lru i was thinking of doing it per device as anyway each device is unlikely to constantly address all the page it has mapped. Simple lru list would do and probably offering some helper for device driver to mark page accessed so page frequently use are not reclaim.
But a global list is fine as well and simplify the case diffirent device use same pages.
Cheers,
Jérôme Glisse
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 16 lines · Benjamin Herrenschmidt
Quoted 6 lines · Jerome Glisse
I do understand that i was pointing out that if i move to, tlb which i am fine with, i will still need to sleep there. That's all i wanted to stress, i did not wanted force using
mmu_notifier, i am fine with them becoming atomic as long as i have a place where i can intercept cpu page table update and propagate them to device mmu.Your MMU notifier can maintain a map of "dirty" PTEs and you do the actual synchronization in the subsequent flush_tlb_* , you need to add hooks there but it's much less painful than in the notifiers.
*However* Linus, even then we can't sleep. We do things like
ptep_clear_flush()that need the PTL and have the synchronous flush semantics.
Quoted 8 lines
Sure, today we wait, possibly for a long time, with IPIs, but we do not sleep. Jerome would have to operate within a similar context. No sleep for you :)
Cheers,
Ben.
So Linus, Benjamin is right there was couple case i did not think about. For instance with cow page, one thread might trigger copy on write allocate new page and update page table and another cpu thread might start using the new page before we even get a chance to update the GPU page table thus GPU could be working on outdated data.
Same kind of race exist on fork when we write protect a page or on when we split a huge page.
I thought that i only needed to special case page reclaimation, migration and forbid things like ksm but i am wrong.
So with that in mind are you ok if i pursue the mmu_notifier case taking into account the result about rwsem+optspin that would allow to make the many fork workload fast while still allowing mmu_notifier callback to sleep ?
Otherwise i have no other choice than to add something like mmu_notifier in the place where there can a be race (huge page split, cow, ...). Which sounds like a bad idea to me when mmu_notifier is perfect for the job.
Cheers,
Jérôme Glisse
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Jerome Glisse
Otherwise i have no other choice than to add something like
mmu_notifierin the place where there can a be race (huge page split, cow, ...). Which sounds like a bad idea to me whenmmu_notifieris perfect for the job.
Even there, how are you going to find a sleepable context ? All that stuff has the PTL held.
Cheers,
Ben.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 3 lines · Jerome Glisse
Otherwise i have no other choice than to add something like
mmu_notifierin the place where there can a be race (huge page split, cow, ...). Which sounds like a bad idea to me whenmmu_notifieris perfect for the job.Even there, how are you going to find a sleepable context ? All that stuff has the PTL held.
Cheers,
Ben.
All i need is invalidate_page and invalidate_range_start both of which are not call while holding any lock beside that anon_vma and or mmap_sem. So i am fine on that front.
The change_pte callback are bracketed by call to invalidate_range_start / end.
What is important is to flush GPU page table prior to any update that would make the cpu page table point to a different page. For invalidate_page this is ok because so far it is call either as page reclaimation and those page is change to swap entry or file entry. Or it is call for page migration and cpu page table is set to special migration entry (there is also the memory failure case but this one is harmless too).
So as far as i can tell i am safe on that front and only mmu_notifier provide the early warning that i need notably for COW.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Linus Torvalds
And as I've mentioned, there is a correct place to slot this in, and that correct way is the _only_ way to ever support future GPU's that _do_ share direct access to the page tables.
The GPU runs a lot faster when using video memory, instead of system memory, on the other side of the PCIe bus.
The CPU cannot directly address video memory.
How can shared page table access work, given these constraints?
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Rik van Riel
The GPU runs a lot faster when using video memory, instead of system memory, on the other side of the PCIe bus.
The nineties called, and they want their old broken model back.
Get with the times. No high-performance future GPU will ever run behind the PCIe bus. We still have a few straggling historical artifacts, but everybody knows where the future is headed.
They are already cache-coherent because flushing caches etc was too damn expensive. They're getting more so.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Rik van Riel
The GPU runs a lot faster when using video memory, instead of system memory, on the other side of the PCIe bus.
Quoted 8 lines
The nineties called, and they want their old broken model back.
Get with the times. No high-performance future GPU will ever run behind the PCIe bus. We still have a few straggling historical artifacts, but everybody knows where the future is headed.
They are already cache-coherent because flushing caches etc was too damn expensive. They're getting more so.
I suppose that VRAM could simply be turned into a very high capacity CPU cache for the GPU, for the case where people want/need an add-on card.
With a few hundred MB of "CPU cache" on the video card, we could offload processing to the GPU very easily, without having to worry about multiple address or page table formats on the CPU side.
A new generation of GPU hardware seems to come out every six months or so, so I guess we could live with TLB invalidations to the first generations of hardware being comically slow :)
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 24 lines · Rik van Riel
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Rik van Riel
The GPU runs a lot faster when using video memory, instead of system memory, on the other side of the PCIe bus.
Quoted 8 lines
The nineties called, and they want their old broken model back.
Get with the times. No high-performance future GPU will ever run behind the PCIe bus. We still have a few straggling historical artifacts, but everybody knows where the future is headed.
They are already cache-coherent because flushing caches etc was too damn expensive. They're getting more so.
I suppose that VRAM could simply be turned into a very high capacity CPU cache for the GPU, for the case where people want/need an add-on card.
With a few hundred MB of "CPU cache" on the video card, we could offload processing to the GPU very easily, without having to worry about multiple address or page table formats on the CPU side.
Quoted 5 lines
A new generation of GPU hardware seems to come out every six months or so, so I guess we could live with TLB invalidations to the first generations of hardware being comically slow :)
I do not want to speak for any GPU manufacturer but i think it is safe to say that there will be dedicated memory for GPU for a long time. It is not going anywhere soon and it is a lot more than few hundred MB, think several GB. If you think about 4k, 8k screen you really gonna want 8GB at least on desktop computer and for compute you will likely see 16GB or 32GB as common size.
Again i stress that there is nothing on the horizon that let me believe that regular memory associated to CPU will ever come close to the bandwith that exist with memory associated to GPU. It is already more than 10 times faster on GPU and as far as i know the gap will grow even more in the next generation.
So dedicated memory to gpu should not be discarded as something that is vanishing quite the contrary it should be acknowledge as something that is here to stay a lot longer afaict.
Cheers,
Jérôme
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
<[email protected]>,Sagi Grimberg <[email protected]>,Shachar Raindel <[email protected]>,Liran Liss <[email protected]>,Roland Dreier <[email protected]>,"Sander, Ben" <[email protected]>,"Stoner, Greg" <[email protected]>,"Bridgman, John" <[email protected]>,"Mantor, Michael" <[email protected]>,"Blinzer, Paul" <[email protected]>,"Morichetti, Laurent" <[email protected]>,"Deucher, Alexander" <[email protected]>,"Gabbay, Oded" <[email protected]>,Davidlohr Bueso <[email protected]>
Message-ID: <[email protected]>
Nothing wrong with device-side memory, but not having it accessible by the CPU seems fundamentally brown from the point of view of unified memory addressing.
Quoted 56 lines · Jerome Glisse
Quoted 2 lines · Rik van Riel
Quoted 1 line · Linus Torvalds
On Tue, May 6, 2014 at 9:30 AM, Rik van Riel <[email protected]>
Quoted 22 lines · wrote:
Quoted 4 lines
Quoted 3 lines
The GPU runs a lot faster when using video memory, instead of system memory, on the other side of the PCIe bus.
Quoted 8 lines
The nineties called, and they want their old broken model back.
Get with the times. No high-performance future GPU will ever run behind the PCIe bus. We still have a few straggling historical artifacts, but everybody knows where the future is headed.
They are already cache-coherent because flushing caches etc was too damn expensive. They're getting more so.
I suppose that VRAM could simply be turned into a very high capacity CPU cache for the GPU, for the case where people want/need an add-on card.
With a few hundred MB of "CPU cache" on the video card, we could offload processing to the GPU very easily, without having to worry about multiple address or page table formats on the CPU side.
Quoted 5 lines
A new generation of GPU hardware seems to come out every six months or so, so I guess we could live with TLB invalidations to the first generations of hardware being comically slow :)
I do not want to speak for any GPU manufacturer but i think it is safe to say that there will be dedicated memory for GPU for a long time. It is not going anywhere soon and it is a lot more than few hundred MB, think several GB. If you think about 4k, 8k screen you really gonna want
8GB at least on desktop computer and for compute you will likely see 16GB or 32GB as common size.Again i stress that there is nothing on the horizon that let me believe that regular memory associated to CPU will ever come close to the bandwith
that exist with memory associated to GPU. It is already more than 10 times
faster on GPU and as far as i know the gap will grow even more in the next
generation.So dedicated memory to gpu should not be discarded as something that is vanishing quite the contrary it should be acknowledge as something that is
here to stay a lot longer afaict.Cheers,
Jérôme
Signature · 7 lines
Sent from my mobile phone. Please pardon brevity and lack of formatting. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · H. Peter Anvin
<[email protected]>,Sagi Grimberg <[email protected]>,Shachar Raindel <[email protected]>,Liran Liss <[email protected]>,Roland Dreier <[email protected]>,"Sander, Ben" <[email protected]>,"Stoner, Greg" <[email protected]>,"Bridgman, John" <[email protected]>,"Mantor, Michael" <[email protected]>,"Blinzer, Paul" <[email protected]>,"Morichetti, Laurent" <[email protected]>,"Deucher, Alexander" <[email protected]>,"Gabbay, Oded" <[email protected]>,Davidlohr Bueso <[email protected]>
Message-ID: <[email protected]>Nothing wrong with device-side memory, but not having it accessible by the CPU seems fundamentally brown from the point of view of unified memory addressing.
Unified memory addressing does not imply CPU and GPU working on same set of data at same time. So having part of the address space only accessible by GPU while it's actively working on it make sense. The GPU then can have low latency (no pcie bus) and enormous bandwith and thus perform the computation a lot faster.
Note that my patchset handle cpu page fault while data is inside GPU memory and migrate it back to system memory. So from CPU point of view it is just as if things were in some kind of swap device except that the swap device is actualy doing some useful computation.
Also on the cache coherent front, cache coherency has a cost, a very high cost. This is why even on APU (where the GPU and CPU are on same die and the mmu of the GPU and CPU have privilege link think today AMD APU or next year intel skylake) you have two memory link, one cache coherent with the CPU and another one that is not cache coherent with the CPU. The latter link is way faster and my patchset is also intended to help taking advantages of this second link (http://developer.amd.com/wordpress/media/2013/06/1004_final.pdf)
Cheers,
Jérôme
Quoted 51 lines · Jerome Glisse
Quoted 49 lines · Rik van Riel
Quoted 2 lines · Linus Torvalds
Quoted 1 line
On Tue, May 6, 2014 at 9:30 AM, Rik van Riel <[email protected]>
Quoted 22 lines · wrote:
Quoted 4 lines
Quoted 3 lines
The GPU runs a lot faster when using video memory, instead of system memory, on the other side of the PCIe bus.
Quoted 8 lines
The nineties called, and they want their old broken model back.
Get with the times. No high-performance future GPU will ever run behind the PCIe bus. We still have a few straggling historical artifacts, but everybody knows where the future is headed.
They are already cache-coherent because flushing caches etc was too damn expensive. They're getting more so.
I suppose that VRAM could simply be turned into a very high capacity CPU cache for the GPU, for the case where people want/need an add-on card.
With a few hundred MB of "CPU cache" on the video card, we could offload processing to the GPU very easily, without having to worry about multiple address or page table formats on the CPU side.
Quoted 5 lines
A new generation of GPU hardware seems to come out every six months or so, so I guess we could live with TLB invalidations to the first generations of hardware being comically slow :)
I do not want to speak for any GPU manufacturer but i think it is safe to say that there will be dedicated memory for GPU for a long time. It is not going anywhere soon and it is a lot more than few hundred MB, think several GB. If you think about 4k, 8k screen you really gonna want
8GB at least on desktop computer and for compute you will likely see 16GB or 32GB as common size.Again i stress that there is nothing on the horizon that let me believe that regular memory associated to CPU will ever come close to the bandwith
that exist with memory associated to GPU. It is already more than 10 times
faster on GPU and as far as i know the gap will grow even more in the next
generation.
Quoted 10 lines
Quoted 7 lines
So dedicated memory to gpu should not be discarded as something that is vanishing quite the contrary it should be acknowledge as something that is
here to stay a lot longer afaict.Cheers,
JérômeSignature · 1 line
Sent from my mobile phone. Please pardon brevity and lack of formatting.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Rik van Riel
Quoted 3 lines
The GPU runs a lot faster when using video memory, instead of system memory, on the other side of the PCIe bus.
Quoted 8 lines
The nineties called, and they want their old broken model back.
Get with the times. No high-performance future GPU will ever run behind the PCIe bus. We still have a few straggling historical artifacts, but everybody knows where the future is headed.
They are already cache-coherent because flushing caches etc was too damn expensive. They're getting more so.
The future might be closer coupled, but it still might not be cache coherent, it might also just be a faster PCIE, considering the current one is a lot faster than the 90s PCI you talk about.
No current high-performance GPU runs in front of the PCIe bus, Intel are still catching up to the performance level of anyone else and others still remain ahead. Even intel make MIC cards for compute that put stuff on the other side of the PCIE divide.
Dave.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Peter Zijlstra
So you forgot to CC Linus, Linus has expressed some dislike for preemptible
mmu_notifiersin the recent past:
I'm glad this came up again.
So I've been running benchmarks (mostly aim7, which nicely exercises our locks) comparing my recent v4 for rwsem optimistic spinning against previous implementation ideas for the anon-vma lock, mostly:
- rwsem (currently)
- rwlock_t
- qrwlock_t
- rwsem+optspin
Of course, *any* change provides significant improvement in throughput for several workloads, by avoiding to block -- there are more performance numbers in the different patches. This is fairly obvious.
What is perhaps not so obvious is that rwsem+optimistic spinning beats all others, including the improved qrwlock from Waiman and Peter. This is mostly because of the idea of cancelable MCS, which was mimic'ed from mutexes. The delta in most cases is around +10-15%, which is non trivial.
I mention this because from a performance PoV, we'll stop caring so much about the type of lock we require in the notifier related code. So while this is not conclusive, I'm not as opposed to keeping the locks blocking as I once was. Now this might still imply things like poor design choices, but that's neither here nor there.
/me sees Sagi smiling ;)
Thanks,
Davidlohr
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Zijlstra
maintainerQuoted 6 lines · Davidlohr Bueso
So I've been running benchmarks (mostly aim7, which nicely exercises our locks) comparing my recent v4 for rwsem optimistic spinning against previous implementation ideas for the anon-vma lock, mostly:
-
rwlock_t
-qrwlock_t
Which reminds me; can you provide the numbers for rwlock_t vs qrwlock_t in a numeric form so I can include them in the qrwlock_t changelog.
That way I can queue those patches for inclusion, I think we want a fair rwlock_t if we can show (and you graphs do iirc) that it doesn't cost us performance.
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Davidlohr Bueso
So I've been running benchmarks (mostly aim7, which nicely exercises our locks) comparing my recent v4 for rwsem optimistic spinning against previous implementation ideas for the anon-vma lock, mostly:
-
rwlock_t
-qrwlock_tWhich reminds me; can you provide the numbers for
rwlock_tvsqrwlock_tin a numeric form so I can include them in theqrwlock_tchangelog.
Ah, right. I was lazy and just showed you the graphs.
Quoted 3 lines
That way I can queue those patches for inclusion, I think we want a fair
rwlock_tif we can show (and you graphs do iirc) that it doesn't cost us performance.
I agree, fairness is much welcome here. And I agree that despite my good numbers, and that we should keep the anon vma lock as a rwsem, its still worth merging the qrwlock stuff. I'll cookup my regular numeric table today.
Thanks,
Davidlohr
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
On Tue, May 6, 2014 at 7:33 PM, Davidlohr Bueso <[email protected]> wrote:
=>
Quoted 5 lines
What is perhaps not so obvious is that rwsem+optimistic spinning beats all others, including the improved qrwlock from Waiman and Peter. This is mostly because of the idea of cancelable MCS, which was mimic'ed from mutexes. The delta in most cases is around +10-15%, which is non trivial.
Ahh, excellent news. That way I don't have to worry about the anonvma lock any more. I'll just forget about it right now, in fact.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 20 lines · Davidlohr Bueso
Quoted 4 lines · Peter Zijlstra
So you forgot to CC Linus, Linus has expressed some dislike for preemptible
mmu_notifiersin the recent past:https://lkml.org/lkml/2013/9/30/385I'm glad this came up again.
So I've been running benchmarks (mostly aim7, which nicely exercises our locks) comparing my recent v4 for rwsem optimistic spinning against previous implementation ideas for the anon-vma lock, mostly:
- rwsem (currently)
-rwlock_t
-qrwlock_t
- rwsem+optspinOf course, *any* change provides significant improvement in throughput for several workloads, by avoiding to block -- there are more performance numbers in the different patches. This is fairly obvious.
Quoted 5 lines
What is perhaps not so obvious is that rwsem+optimistic spinning beats all others, including the improved qrwlock from Waiman and Peter. This is mostly because of the idea of cancelable MCS, which was mimic'ed from mutexes. The delta in most cases is around +10-15%, which is non trivial.
These are great news David!
Quoted 5 lines
I mention this because from a performance PoV, we'll stop caring so much about the type of lock we require in the notifier related code. So while this is not conclusive, I'm not as opposed to keeping the locks blocking as I once was. Now this might still imply things like poor design choices, but that's neither here nor there.
So is the rwsem+opt strategy the way to go Given it keeps everyone happy? We will be more than satisfied with it as it will allow us to guarantee device MMU update.
Quoted 1 line
/me sees Sagi smiling ;)
:)
Sagi.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 35 lines · sagi grimberg
Quoted 20 lines · Davidlohr Bueso
Quoted 4 lines · Peter Zijlstra
So you forgot to CC Linus, Linus has expressed some dislike for preemptible
mmu_notifiersin the recent past:I'm glad this came up again.
So I've been running benchmarks (mostly aim7, which nicely exercises our locks) comparing my recent v4 for rwsem optimistic spinning against previous implementation ideas for the anon-vma lock, mostly:
- rwsem (currently)
-rwlock_t
-qrwlock_t
- rwsem+optspinOf course, *any* change provides significant improvement in throughput for several workloads, by avoiding to block -- there are more performance numbers in the different patches. This is fairly obvious.
Quoted 5 lines
What is perhaps not so obvious is that rwsem+optimistic spinning beats all others, including the improved qrwlock from Waiman and Peter. This is mostly because of the idea of cancelable MCS, which was mimic'ed from mutexes. The delta in most cases is around +10-15%, which is non trivial.
These are great news David!
Quoted 5 lines
I mention this because from a performance PoV, we'll stop caring so much about the type of lock we require in the notifier related code. So while this is not conclusive, I'm not as opposed to keeping the locks blocking as I once was. Now this might still imply things like poor design choices, but that's neither here nor there.
Quoted 8 lines
So is the rwsem+opt strategy the way to go Given it keeps everyone happy? We will be more than satisfied with it as it will allow us to guarantee device
MMU update.Quoted 1 line
/me sees Sagi smiling ;)
:)
So i started doing thing with tlb flush but i must say things looks ugly. I need a new page flag (goodbye 32bits platform) and i need my own lru and page reclaimation for any page in use by a device, i need to hook up inside try_to_unmap or migrate (but i will do the former). I am trying to be smart by trying to schedule a worker on another cpu before before sending the ipi so that while the ipi is in progress hopefully another cpu might schedule the invalidation on the GPU and the wait after ipi for the gpu will be quick.
So all in all this is looking ugly and it does not change the fact that i sleep (well need to be able to sleep). It just move the sleeping to another part.
Maybe i should stress that with the mmu_notifier version it only sleep for process that are using the GPU those process are using userspace API like OpenCL which are not playing well with fork, ie read do not use fork if you are using such API.
So for my case if a process has mm->hmm set to something that would mean that there is a GPU using that address space and that it is unlikely to go under the massive workload that people try to optimize the anon_vma lock for.
My point is that with rwsem+optspin it could try spinning if mm->hmm was NULL and make the massive fork workload go fast, or it could sleep directly if mm->hmm is set.
This way my addition are not damaging anyone workload, only the workload that would use hmm would likely have lock contention on fork but those workload should not fork in the first place and if they do they should pay a price.
I will finish up the tlb hackish version of hmm so people can judge how ugly it is (in my view) and send it here as soon as i can.
But i think it's clear that with rwsem+optspin we can make all workload happy and fast.
Cheers,
Jérôme Glisse
Quoted 2 lines
Sagi.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 62 lines · Jerome Glisse
Quoted 35 lines · sagi grimberg
Quoted 20 lines · Davidlohr Bueso
Quoted 4 lines · Peter Zijlstra
So you forgot to CC Linus, Linus has expressed some dislike for preemptible
mmu_notifiersin the recent past:I'm glad this came up again.
So I've been running benchmarks (mostly aim7, which nicely exercises our locks) comparing my recent v4 for rwsem optimistic spinning against previous implementation ideas for the anon-vma lock, mostly:
- rwsem (currently)
-rwlock_t
-qrwlock_t
- rwsem+optspinOf course, *any* change provides significant improvement in throughput for several workloads, by avoiding to block -- there are more performance numbers in the different patches. This is fairly obvious.
Quoted 5 lines
What is perhaps not so obvious is that rwsem+optimistic spinning beats all others, including the improved qrwlock from Waiman and Peter. This is mostly because of the idea of cancelable MCS, which was mimic'ed from mutexes. The delta in most cases is around +10-15%, which is non trivial.
These are great news David!
Quoted 5 lines
I mention this because from a performance PoV, we'll stop caring so much about the type of lock we require in the notifier related code. So while this is not conclusive, I'm not as opposed to keeping the locks blocking as I once was. Now this might still imply things like poor design choices, but that's neither here nor there.
Quoted 8 lines
So is the rwsem+opt strategy the way to go Given it keeps everyone happy? We will be more than satisfied with it as it will allow us to guarantee device
MMU update.Quoted 1 line
/me sees Sagi smiling ;)
:)
So i started doing thing with tlb flush but i must say things looks ugly. I need a new page flag (goodbye 32bits platform) and i need my own lru and page reclaimation for any page in use by a device, i need to hook up inside
try_to_unmapor migrate (but i will do the former). I am trying to be smart by trying to schedule a worker on another cpu before before sending the ipi so that while the ipi is in progress hopefully another cpu might schedule the invalidation on the GPU and the wait after ipi for the gpu will be quick.So all in all this is looking ugly and it does not change the fact that i sleep (well need to be able to sleep). It just move the sleeping to another part.
Maybe i should stress that with the
mmu_notifierversion it only sleep for process that are using the GPU those process are using userspace API like OpenCL which are not playing well with fork, ie read do not use fork if you are using such API.
Quoted 8 lines
So for my case if a process has
mm->hmmset to something that would mean that there is a GPU using that address space and that it is unlikely to go under the massive workload that people try to optimize theanon_vmalock for.My point is that with rwsem+optspin it could try spinning if
mm->hmmwas NULL and make the massive fork workload go fast, or it could sleep directly ifmm->hmmis set.
Sorry? Unless I'm misunderstanding you, we don't do such things. Our locks are generic and need to work for any circumstance, no special cases here and there... _specially_ with these kind of things. So no, rwsem will spin as long as the owner is set, just like any other users.
Thanks,
Davidlohr
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 72 lines · Davidlohr Bueso
Quoted 62 lines · Jerome Glisse
Quoted 35 lines · sagi grimberg
Quoted 20 lines · Davidlohr Bueso
Quoted 4 lines · Peter Zijlstra
So you forgot to CC Linus, Linus has expressed some dislike for preemptible
mmu_notifiersin the recent past:I'm glad this came up again.
So I've been running benchmarks (mostly aim7, which nicely exercises our locks) comparing my recent v4 for rwsem optimistic spinning against previous implementation ideas for the anon-vma lock, mostly:
- rwsem (currently)
-rwlock_t
-qrwlock_t
- rwsem+optspinOf course, *any* change provides significant improvement in throughput for several workloads, by avoiding to block -- there are more performance numbers in the different patches. This is fairly obvious.
Quoted 5 lines
What is perhaps not so obvious is that rwsem+optimistic spinning beats all others, including the improved qrwlock from Waiman and Peter. This is mostly because of the idea of cancelable MCS, which was mimic'ed from mutexes. The delta in most cases is around +10-15%, which is non trivial.
These are great news David!
Quoted 5 lines
I mention this because from a performance PoV, we'll stop caring so much about the type of lock we require in the notifier related code. So while this is not conclusive, I'm not as opposed to keeping the locks blocking as I once was. Now this might still imply things like poor design choices, but that's neither here nor there.
Quoted 8 lines
So is the rwsem+opt strategy the way to go Given it keeps everyone happy? We will be more than satisfied with it as it will allow us to guarantee device
MMU update.Quoted 1 line
/me sees Sagi smiling ;)
:)
So i started doing thing with tlb flush but i must say things looks ugly. I need a new page flag (goodbye 32bits platform) and i need my own lru and page reclaimation for any page in use by a device, i need to hook up inside
try_to_unmapor migrate (but i will do the former). I am trying to be smart by trying to schedule a worker on another cpu before before sending the ipi so that while the ipi is in progress hopefully another cpu might schedule the invalidation on the GPU and the wait after ipi for the gpu will be quick.So all in all this is looking ugly and it does not change the fact that i sleep (well need to be able to sleep). It just move the sleeping to another part.
Maybe i should stress that with the
mmu_notifierversion it only sleep for process that are using the GPU those process are using userspace API like OpenCL which are not playing well with fork, ie read do not use fork if you are using such API.Quoted 8 lines
So for my case if a process has
mm->hmmset to something that would mean that there is a GPU using that address space and that it is unlikely to go under the massive workload that people try to optimize theanon_vmalock for.My point is that with rwsem+optspin it could try spinning if
mm->hmmwas NULL and make the massive fork workload go fast, or it could sleep directly ifmm->hmmis set.
Quoted 8 lines
Sorry? Unless I'm misunderstanding you, we don't do such things. Our locks are generic and need to work for any circumstance, no special cases here and there... _specially_ with these kind of things. So no, rwsem will spin as long as the owner is set, just like any other users.
Thanks,
Davidlohr
I do not mind spining all time i was just thinking that it could be optimize away in case there is hmm for the current mm as it means that any way there very much likely gonna be a schedule inside the mmu_notifier.
But if you prefer keep code generic i am fine with wasting cpu cycle.
Cheers,
Jérôme Glisse
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>